Image processing method, device, electronic device and storage medium

Abstract

The present invention discloses an image processing method, device, electronic device and storage medium. The method comprises: collecting images to be processed corresponding to the target subject at the current moment and the previous moment; processing each image to be processed based on a target posture estimation model to obtain the posture information of the target subject at the next moment; wherein the target posture estimation model comprises a feature extraction model and a target posture attribute determination model, the target posture estimation model is determined based on a pre-constructed training sample set, the training sample set comprises multiple training samples, the training samples comprise a spliced ​​image and a posture label, the spliced ​​image is spliced ​​based on the current video frame and the previous video frame, and the posture label corresponds to the next video frame. The method solves the problem in the prior art that the posture is estimated by extracting geometric constraints from a single frame image based on a geometric algorithm, resulting in low posture estimation accuracy, and achieves the effect of improving the accuracy and convenience of posture estimation.

Description

Technical Field

[0001] The present invention relates to the field of computer processing technology, and in particular to an image processing method, device, electronic equipment and storage medium. Background Art

[0002] In the odometer positioning requirements, it is crucial to accurately and efficiently measure the trajectory of a moving object. An important feature of the odometer is that it only cares about the movement in local time. When time is sampled at a certain interval, the posture of the moving object in each time interval can be estimated. Because the estimation is affected by noise, the estimation error at the previous moment will be accumulated to the movement at the later moment, that is, the "drift" phenomenon occurs. In recent years, visual odometer positioning methods have emerged as a supplement to GPS, inertial navigation systems, wheel odometers, etc. The key issue is to perform posture estimation and robot positioning based on the captured images. Monocular vision calculation in visual odometers is simple and stable, and is widely used.

[0003] Currently, most of the most advanced visual odometry methods are based on geometry-based algorithms that estimate pose by extracting geometric constraints from a single frame of image. This method has low pose accuracy. Summary of the invention

[0004] The present invention provides an image processing method, device, electronic device and storage medium to achieve the technical effect of improving the accuracy and convenience of posture estimation.

[0005] According to one aspect of the present invention, there is provided an image processing method, the method comprising:

[0006] Collect the images to be processed corresponding to the target subject at the current moment and the previous moment;

[0007] Processing each image to be processed based on the target posture estimation model to obtain the posture information of the target subject at the next moment;

[0008] Among them, the target posture estimation model includes a feature extraction model and a target posture attribute determination model, the target posture estimation model is determined based on a pre-constructed training sample set, the training sample set includes multiple training samples, the training samples include spliced ​​images and posture labels, the spliced ​​image is based on the splicing of the current video frame and the previous video frame, and the posture label corresponds to the next video frame of the current video frame.

[0009] According to another aspect of the present invention, there is provided an image processing device, the device comprising:

[0010] The to-be-processed image acquisition module is used to acquire the to-be-processed images corresponding to the target subject at the current moment and the previous moment;

[0011] A posture information determination module is used to process each image to be processed based on a target posture estimation model to obtain the posture information of the target subject at the next moment;

[0012] Among them, the target posture estimation model includes a feature extraction model and a target posture attribute determination model, the target posture estimation model is determined based on a pre-constructed training sample set, the training sample set includes multiple training samples, the training samples include spliced ​​images and posture labels, the spliced ​​image is based on the splicing of the current video frame and the previous video frame, and the posture label corresponds to the next video frame of the current video frame.

[0013] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the image processing method described in any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the image processing method described in any embodiment of the present invention when executed.

[0018] The technical solution of the embodiment of the present invention is to collect the images to be processed corresponding to the target subject at the current moment and the previous moment, and process each image to be processed based on the target posture estimation model to obtain the posture information of the target subject at the next moment; wherein the target posture estimation model includes a feature extraction model and a target posture attribute determination model, the target posture estimation model is determined based on a pre-constructed training sample set, the training sample set includes multiple training samples, the training samples include a spliced ​​image and a posture label, the spliced ​​image is based on the current video frame and the previous video frame, and the posture label corresponds to the next video frame, which solves the problem of low posture estimation accuracy caused by extracting geometric constraints from a single frame image based on a geometric algorithm in the prior art, and realizes pre-construction of the target posture estimation model by using the spliced ​​image spliced ​​from the current video frame and the previous video frame and the posture label corresponding to the next video frame, when the images to be processed corresponding to the target subject at the current moment and the previous moment are collected, each image to be processed is processed based on the target posture estimation model to obtain the posture information of the target subject at the next moment, thereby achieving the technical effect of improving the accuracy and convenience of posture estimation.

[0019] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 is a flowchart of an image processing method provided according to Embodiment 1 of the present invention;

[0022] Figure 2 is a flowchart of an image processing method provided according to Embodiment 2 of the present invention;

[0023] Figure 3 This is a U-Net network structure diagram provided according to Embodiment 2 of the present invention;

[0024] Figure 4 1 is a diagram of an LSTM-based RNN network architecture according to Embodiment 2 of the present invention;

[0025] Figure 5 This is an internal structure diagram of the LSTM provided according to the second embodiment of the present invention;

[0026] Figure 6 is a flowchart of an image processing method provided according to Embodiment 3 of the present invention;

[0027] Figure 7 is a structural schematic diagram of an image processing device provided according to a fourth embodiment of the present invention;

[0028] Figure 8 It is a schematic diagram of the structure of an electronic device for implementing the image processing method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] Embodiment 1

[0032] Figure 1 is a flowchart of an image processing method provided according to the first embodiment of the present invention. This embodiment is applicable to the case of posture estimation. The method can be executed by an image processing device. The image processing device can be implemented in the form of hardware and / or software. The XX device can be configured in a computing device. Figure 1 As shown, the method includes:

[0033] S110, collecting the images to be processed corresponding to the target subject at the current moment and the previous moment.

[0034] The image to be processed can be understood as an image that needs to be processed. This image can be an image captured by a camera. In practical applications, a monocular camera can be deployed on the target subject. After the monocular camera is used to capture an image, the posture (such as displacement or rotation angle) of the target subject can be estimated based on the captured image. The target subject can be a vehicle or a robot.

[0035] In this embodiment, images can be captured in real time based on a camera device deployed in the target subject. At this time, the image at the current moment and the image captured at the previous moment can be obtained as images to be processed, so as to estimate the posture of the target subject at the next moment based on these two images.

[0036] S120, processing each image to be processed based on the target posture estimation model to obtain posture information of the target subject at the next moment.

[0037] The target posture estimation model includes a feature extraction model and a target posture attribute determination model. The feature extraction model can be a U-Net network for image feature extraction. The target posture attribute determination model can be an RNN network for image feature recognition. The target posture estimation model is determined based on a pre-constructed training sample set, which includes multiple training samples. The training samples include a spliced ​​image and a posture label. The spliced ​​image is based on the splicing of the current video frame and the previous video frame, and the posture label corresponds to the next video frame of the current video frame.

[0038] Specifically, the target posture estimation model can identify and process the features in the to-be-processed images at the current moment and the previous moment to obtain the posture information of the target subject at the next moment. Optionally, the posture information can include displacement and rotation angle.

[0039] The technical solution of this embodiment is to collect the images to be processed corresponding to the target subject at the current moment and the previous moment, and process each image to be processed based on the target posture estimation model to obtain the posture information of the target subject at the next moment; wherein the target posture estimation model includes a feature extraction model and a target posture attribute determination model, and the target posture estimation model is determined based on a pre-constructed training sample set, and the training sample set includes multiple training samples, and the training samples include a spliced ​​image and a posture label. The spliced ​​image is based on the current video frame and the previous video frame, and the posture label corresponds to the next video frame. This solves the problem of low posture estimation accuracy caused by extracting geometric constraints from a single frame image based on a geometric algorithm in the prior art, and realizes that the target posture estimation model is pre-constructed by using the spliced ​​image based on the current video frame and the previous video frame and the posture label corresponding to the next video frame. When the images to be processed corresponding to the target subject at the current moment and the previous moment are collected, each image to be processed is processed based on the target posture estimation model to obtain the posture information of the target subject at the next moment, thereby achieving the technical effect of improving the accuracy and convenience of posture estimation.

[0040] Embodiment 2

[0041] Figure 2 1 is a flowchart of an image processing method provided according to Embodiment 2 of the present invention. On the basis of the above-mentioned embodiment, a target posture estimation model can also be obtained by pre-training based on a training sample set. The specific implementation method can refer to the technical solution of this embodiment. Among them, the technical terms that are the same as or corresponding to the above-mentioned embodiment are not repeated here.

[0042] like Figure 2 As shown, the method specifically comprises the following steps:

[0043] S210: Determine each training sample in the training sample set.

[0044] It should be noted that before training the target posture estimation model, it is necessary to first obtain training sample data so as to train based on the training sample data. In order to improve the accuracy of the model, as many and rich training samples as possible can be obtained.

[0045] The samples used in training the target pose estimation model can be used as training samples. The training samples include a spliced ​​image and a pose label. The spliced ​​image is based on the splicing of the current video frame and the previous video frame, and the pose label corresponds to the next video frame of the current video frame.

[0046] In this embodiment, the determination of each training sample in the training sample set includes: collecting multiple continuous video frames and determining the grayscale value to be used corresponding to each video frame; determining the grayscale mean based on each grayscale value to be used; for each video frame, determining the target grayscale value of the current video frame based on the grayscale mean and the grayscale value to be used of the current video frame; determining each training sample based on the target grayscale value and posture label corresponding to each video frame.

[0047] Among them, the video frame can be an image captured by a camera device, or it can be an image pre-stored in a storage space. For example, a monocular camera device can be deployed in the target subject in advance, and video can be captured in real time during the movement of the target subject. The video frames in the video are continuous, and each video frame is an RGB (Red Green Blue) image. The monocular RGB image sequence can also be used as a plurality of continuous video frames. The processing method for determining the target grayscale value of each video frame is the same, and determining the target grayscale value of any video frame is used as an example to determine the target grayscale value of the current video frame. The grayscale value can be an RGB value.

[0048] In this embodiment, a plurality of continuous video frames shot by a monocular camera device can be collected from the Internet or in an actual scene, and the grayscale value of each video frame can be determined as the grayscale value to be used. The grayscale values ​​to be used can be averaged to obtain the average grayscale value of all video frames, that is, the grayscale mean. The grayscale value of the current video frame can be subtracted from the grayscale mean to obtain the target grayscale value of the current video frame. Exemplarily, in order to enable the network to learn effective feature representations and reduce the dimensionality, a clipped video or monocular RGB image sequence can be used as a data set. Let the data set X = {x1, x2, ..., x N}, where the dataset contains N RGB images, i.e., video frames. The average RGB value of all video frames in dataset X, i.e., the grayscale mean is At each time t, subtract the grayscale mean from the RGB value of the current video frame The target grayscale value of the current video frame can be obtained, that is, Furthermore, the size of each video frame can be resized to 1404*508 so that each video frame can adapt to the U-Net network structure, that is, the feature extraction model. The training samples can be determined based on the target grayscale value and posture label corresponding to each video frame. For example, the resized current video frame can be spliced ​​with the resized previous video frame based on the target grayscale value to obtain a spliced ​​image, and the spliced ​​image and the posture label of the next video frame of the current video frame can be used as a training sample.

[0049] In order to enable the model to learn knowledge from the image context information, when determining each training sample based on the target grayscale value and posture label corresponding to each video frame, the input of the model can be determined based on the two connected video frames in the sequence frame, and the output of the model can be determined based on the posture label of the video frame after these two video frames, so that the training samples can be determined based on the context information of the sequence frame, thereby improving the accuracy of the model in estimating the posture of the target subject at the next moment.

[0050] Optionally, each training sample is determined based on the target grayscale value and posture label corresponding to each video frame, including: for each video frame, based on the target grayscale values ​​corresponding to the current video frame and the previous video frame, determining the stitched image corresponding to the current video frame; based on the stitched image corresponding to the corresponding video frame and the posture label of the next video frame corresponding to the video frame, determining the training sample.

[0051] It should be noted that the processing method for determining the stitched image corresponding to each video frame is the same, and the determination of the stitched image corresponding to any video frame is used as the determination of the stitched image corresponding to the current video frame for illustration.

[0052] In this embodiment, after the size of the target grayscale value corresponding to each video frame is adapted to the U-Net network structure, the target grayscale value corresponding to the current video frame and the previous video frame can be spliced ​​to obtain an image after the two video frames are spliced ​​as a spliced ​​image. For example, two consecutive RGB video frames can be cascaded to form a new image tensor, i.e., a spliced ​​image. The spliced ​​image and the posture label corresponding to the next video frame of the current video frame can be used as a training sample, and accordingly, multiple training samples can be obtained.

[0053] S220, training a posture estimation model to be trained based on each training sample to obtain the target posture estimation model, and processing the images to be processed of the target subject at the current moment and the previous moment based on the target posture estimation model to obtain the posture information of the target subject at the next moment.

[0054] It should be noted that the target posture estimation model is a model obtained after the training sample is processed by the training posture estimation model. After obtaining the training sample, the training posture estimation model can be trained based on the spliced ​​images and posture labels in each training sample to obtain the target posture estimation model. The target posture estimation model can be deployed in the target subject, so that when the images to be processed at the current moment and the previous moment are collected, each collected image to be processed can be processed to obtain the posture information of the target subject at the next moment of the current moment.

[0055] It can be understood as: taking the spliced ​​image as the input of the pose estimation model to be trained, and taking the pose label corresponding to the spliced ​​image as the model output, to train the target pose estimation model.

[0056] It should also be noted that the target posture estimation model includes a feature extraction model and a target posture attribute determination model. Accordingly, when the posture estimation model to be trained is trained based on each training sample to obtain the target posture estimation model, each training sample can be used as the input of the feature extraction model, the output of the feature extraction model can be used as the input of the posture attribute determination model to be trained, and the posture label corresponding to the spliced ​​image can be used as the output of the posture attribute determination model to be trained to obtain the target posture estimation model. Among them, the target posture attribute determination model is a model obtained after the training sample processed by the feature extraction model processes the posture attribute determination model to be trained.

[0057] Optionally, the posture estimation model to be trained is trained based on each training sample to obtain the target posture estimation model, including: for each training sample, the spliced ​​image in the current training sample is processed based on the feature extraction model to obtain a target feature map corresponding to the spliced ​​image, and the target feature map is used as the input of the posture attribute determination model to be trained, and the corresponding posture label is used as the output of the posture attribute determination model to be trained to train the posture attribute determination model to be trained; the convergence of the loss function in the posture attribute determination model to be trained is used as the training target to obtain the target posture attribute determination model; the feature extraction model and the target posture attribute determination model are spliced ​​to obtain the target posture estimation model.

[0058] It should be noted that the processing method for each training sample is the same, and the processing of one of the training samples is used as an example for explanation. The convergence of the preset loss function can be used as the training goal. When it is determined that the preset loss function of the posture attribute determination model to be trained converges, it means that this model can be used as the target posture attribute determination model.

[0059] In this embodiment, the spliced ​​image in the current training sample can be input into the feature extraction model, the feature extraction model can extract geometric features from the spliced ​​image, and output the target feature map corresponding to the spliced ​​image. The target feature map can be used as the input of the posture attribute determination model to be trained, and the corresponding posture label can be used as the output of the posture attribute determination model to be trained. Since the model parameters in the posture attribute determination model to be trained are not corrected, then, the posture information output by the model at this time may also be different from the posture label corresponding to the current training sample. The model parameters in the posture attribute determination model to be trained can be continuously corrected to obtain a trained target posture attribute determination model. The feature extraction model and the target posture attribute determination model can be cascaded to obtain a target posture estimation model. When the collected images to be processed at the current moment and the previous moment are input into the target posture estimation model, the target posture estimation model can output the posture label expected at the next moment.

[0060] It should be noted that the feature extraction model can be a U-Net network, which is a network structure that includes downsampling and upsampling processes. The downsampling process is used to gradually display geometric (environmental) information. The continuous deepening of convolution will cause the loss of a lot of important information in the image, while the upsampling process can strengthen the network's extraction of effective features and gradually restore the accuracy of the image by combining the geometric information of each layer in the downsampling and the input information in the upsampling. Optionally, the stitched image in the current training sample is processed based on the feature extraction model to obtain a target feature map corresponding to the stitched image, including: processing the stitched image based on a first preset number of convolutional layers at a first sampling rate to obtain a first feature map to be processed, and downsampling the first feature map to be processed based on a second sampling rate to obtain a first feature map to be used; processing the first feature map to be used based on a second preset number of convolutional layers at the first sampling rate to obtain a second feature map to be processed, and downsampling the second feature map to be processed based on the second sampling rate to obtain a second feature map to be used; processing the second feature map to be used based on a third preset number of convolutional layers at the first sampling rate to obtain a third feature map to be processed, and downsampling the third feature map to be processed based on the second sampling rate to obtain a third feature map to be used; processing the third feature map to be used based on a fourth preset number of convolutional layers at the first sampling rate to obtain a fourth feature map to be processed, and downsampling the fourth feature map to be processed based on the second sampling rate to obtain a fourth feature map to be used, and upsampling the fourth feature map to be used to obtain a target feature map.

[0061] The first preset number may be 64, the second preset number may be 128, the third preset number may be 256, the fourth preset number may be 512, and the fifth preset number may be 1024. The first sampling rate may be a convolution operation with a convolution kernel size of 3*3 and an activation function of ReLU. The second sampling rate may be a maximum pooling operation with a convolution kernel size of 2*2 and a stride of 2.

[0062] In this embodiment, reference can be made to Figure 3 The U-Net network structure diagram shown in the figure is used for explanation. The U-Net network has a total of four layers, and the spliced ​​image is downsampled 4 times and upsampled 4 times respectively. The spliced ​​image can be input into the U-Net network (i.e., the feature extraction model). First, the spliced ​​image can be processed using the first layer of the U-Net network. For example, the spliced ​​image can be processed using a first preset number (e.g., 64) of convolution layers according to a first sampling rate to obtain a first feature map to be processed. For example, the input spliced ​​image is a 1404*508 6-dimensional image. The spliced ​​image is convolved with 64 3×3 convolution kernels, and then 64 1402*506 feature channels are obtained after passing through the ReLU function. Then, the result of 1402*502×64 is convolved with 64 3×3 convolution kernels, and 64 1400*504 feature extraction results are obtained after passing through the ReLU function. This is the feature image of 1400*504×64, which is the processing result of the first layer. That is, the first feature map to be processed is obtained. Further, the first feature map to be processed is downsampled based on the second sampling rate to obtain the first feature map to be used. For example, the first feature map to be processed is downsampled to half of its original size: 700*252 through a 2*2 convolution kernel and a maximum pooling operation with a step size of 2, and then the image features are further extracted through 128 convolution kernels to obtain the first feature map to be used. The subsequent downsampling process is also analogous. Each layer will undergo two convolutions to extract image features; each downsampling layer will reduce the image by half and double the number of convolution kernels. The final result of the downsampling part is 84×28×1024 (the fourth feature map to be used), that is, there are a total of 1024 feature layers, and the feature size of each layer is 80×24. Further, the fourth feature map to be used can be upsampled to obtain the target feature map.

[0063] It should be noted that in order to improve the accuracy of feature map acquisition, during the upsampling process, the feature map that needs to be upsampled in each layer can be spliced ​​with the feature map that needs to be downsampled in the layer, so that the spliced ​​feature map can be upsampled to improve the accuracy of the target feature map and further improve the model training accuracy.

[0064] Optionally, upsampling the fourth feature map to be used to obtain a target feature map includes: upsampling the fourth feature map to be used based on the third sampling rate to obtain a fourth feature map to be spliced, and splicing the fourth feature map to be spliced ​​with the sheared fourth feature map to be processed to obtain a fourth feature map to be convolved; processing the fourth feature map to be convolved according to the first sampling rate based on a fourth preset number of convolutional layers to obtain a fourth feature map to be sampled, and upsampling the fourth feature map to be sampled based on the third sampling rate to obtain a third feature map to be spliced, and splicing the third feature map to be spliced ​​with the sheared third feature map to be processed to obtain a third feature map to be convolved; processing the third feature map to be convolved according to the first sampling rate based on the third preset number of convolutional layers The method further comprises the steps of: processing the second feature map to be convolved according to the first sampling rate based on the second preset number of convolutional layers to obtain a second feature map to be sampled, and upsampling the third feature map to be sampled based on the third sampling rate to obtain a second feature map to be spliced, and splicing the second feature map to be spliced ​​with the sheared second feature map to be processed to obtain a second feature map to be convolved; processing the second feature map to be convolved according to the first sampling rate based on the second preset number of convolutional layers to obtain a second feature map to be sampled, and upsampling the second feature map to be sampled based on the third sampling rate to obtain a first feature map to be spliced; splicing the first feature map to be spliced ​​with the sheared first feature map to be processed to obtain a first feature map to be convolved, and processing the first feature map to be convolved according to the first sampling rate based on the first preset number of convolutional layers to obtain a target feature map.

[0065] The third sampling rate may be a deconvolution operation with a convolution kernel of 2*2.

[0066] In this embodiment, continue to see Figure 3, the right part is 4 upsampling processes from bottom to top. Starting from the bottom right corner, the 84×28×1024 feature matrix is ​​deconvolved through 512 2*2 convolution kernels, and the matrix is ​​expanded to 160×48×512, which is the fourth feature map to be spliced. Since deconvolution can only expand the image but not restore it, in order to reduce data loss, the left downsampled image is cropped to the same size and then directly spliced ​​together to increase the feature layer, and the fourth feature map to be convolved is 160×48×1024. Then, the fourth feature map to be convolved is extracted based on the maximum pooling operation of the second sampling rate, and the fourth feature map to be sampled is 156×44×512. Since each downsampling will make the result smaller, the feature map in the left downsampling process needs to be cropped before each splicing, and then the matrix is ​​spliced. The entire new feature matrix becomes 312×88×512 (i.e., the third feature map to be convolved), and then after two convolutions with 256 convolution kernels, a feature matrix of 308×84×256 is obtained (i.e., the third feature map to be sampled), and then upsampling is performed again, and the above process is repeated. Each layer will perform two convolutions to extract features. Each upsampling layer will double the image and reduce the number of convolution kernels by half. The final upsampling result is 1220×324×64 (i.e., the target feature map), which means there are 64 feature layers in total, and the feature size of each layer is 1220×324.

[0067] On the basis of the above technical solution, the target feature map can be input into the posture attribute determination model to be trained. It should be noted that in order to discover and utilize the correlation between the image sequences taken on a long trajectory and enhance the ability to learn image context information, LSTM (Long Short-Term Memory) in RNN that can learn long-term dependencies can be used. LSTM can explicitly discard or retain the previous hidden state to update the current state in order to estimate the posture during the learning process.

[0068] Optionally, the target feature map is used as the input of the posture attribute determination model to be trained, and the corresponding posture label is used as the output of the posture attribute determination model to be trained, and the posture attribute determination model to be trained is trained, including: cascading multiple posture memory models to be used to construct the posture attribute determination model to be trained; processing the target feature map based on each posture memory model to be used to obtain posture data to be used; and training the posture attribute determination model to be trained based on the error value between the posture data to be used and the posture label.

[0069] In order to learn high-level feature representation and realize dynamic modeling, an RNN network can be constructed by stacking three LSTM layers, that is, the posture attribute determination model to be trained, which can be seen in Figure 4The LSTM-based RNN network architecture diagram is shown in the figure. The posture attribute determination model to be trained includes LSTM1, LSTM2 and LSTM3. Each LSTM layer is used as a posture memory model to be used. The hidden state of one LSTM is the input of another LSTM. Each LSTM layer contains 800 hidden states.

[0070] In this embodiment, the posture attribute determination model to be trained constructed by cascading multiple LSTM layers (i.e., the posture memory model to be used) can be trained, and the target feature map at time t can be used as the input of the first posture memory model to be used (LSTM1), and the target feature map can be processed based on the posture memory model to be used. For example, see Figure 5 The internal structure diagram of the LSTM shown in Figure 1 shows that each LSTM is associated with a time step. Given the input x at time t t , hidden state h t-1 and the memory unit m of the previous LSTM t-1 , the update rule of LSTM at the current time t is as follows:

[0071] i t =σ(W xi x t +W hi h t-1 +b i )

[0072] f t =σ(W xf x t +W hf h t-1 +b f )

[0073] g t =tanh(W xg x t +W hg h t-1 +b g )

[0074] m t =f t ⊙m t-1 +i t ⊙g t

[0075] o t =σ(W xo x t +W ho h t-1 +b o )

[0076] h t=o t ⊙tanh(m t )

[0077] Among them, ⊙ represents the element-wise product of two vectors; σ represents the sigmoid nonlinear function; tanh is the hyperbolic tangent nonlinear function; W represents the corresponding weight matrix; b represents the bias vector; i t 、f t , g t 、m t and t They correspond to the input gate, forget gate, input modulation gate, memory unit and output gate at time t respectively. The hidden state of LSTM1 can be used as the input of LSTM2, and the hidden state of LSTM2 can be used as the input of LSTM3. Through the processing of LSTM3, the posture data to be used is obtained, so that the RNN network can output a posture estimate at each moment according to the effective features generated by U-Net, so that the posture estimate develops over time as the camera moves and captures images.

[0078] On the basis of the above technical solution, the posture attribute determination model to be trained can be trained based on the error value between the posture data to be used and the posture label. In this embodiment, the error value can be determined based on the posture information output by the posture attribute determination model to be trained and the corresponding posture label, and then the model parameters in the posture attribute determination model to be trained can be corrected based on the error value. Exemplarily, a set of monocular RGB image sequences X ending at time t can be given. t ={x1,x2,...x t}, the visual odometer system based on U-Net and RNN calculates the conditional probability Y of the posture t ={y1,y2,...,y t}as follows:

[0079] p(Y t |X t )=p(y1,…,y t |x1,…,x t )

[0080] In order to maximize the conditional probability, a parameter θ is introduced so that θ * =argmaxp(Y t |X t ; θ), in order to learn the hyperparameter θ, this technical solution minimizes the posture label at time t Pose information estimated by the model The difference between them is the optimization target, and the displacement at time t is obtained. The loss function is shown in formula (1):

[0081]

[0082] Get the rotation angle (also called Euler angle) at time t The loss function is shown in formula (2):

[0083]

[0084] The final loss function of pose estimation is shown in formula (3):

[0085]

[0086] Among them, α is a scaling factor used to balance the weights of displacement and rotation angle losses; N is the number of samples. The loss function of the final model is shown in formula (4):

[0087]

[0088] The training error of the loss function (formula (4)), i.e., the loss parameter, can be used as a condition for detecting whether the loss function has reached convergence, such as whether the training error is less than a preset error or whether the error change trend tends to be stable, or whether the current number of iterations is equal to the preset number. If the detection reaches the convergence condition, such as the training error of the loss function reaches less than the preset error or the error change tends to be stable, it indicates that the training of the posture attribute determination model to be trained is completed, and the iterative training can be stopped at this time. If it is detected that the convergence condition is not reached at present, further training samples can be obtained and processed by the feature extraction model to determine the target feature map, and the posture attribute determination model to be trained is continued to be trained until the training error of the loss function is within the preset range. When the training error of the loss function reaches convergence, the posture attribute determination model to be trained can be used as the target posture attribute determination model, and the feature extraction model and the target posture attribute determination model can be cascaded to obtain the target posture estimation model. So that when the collected images to be processed at the current moment and the previous moment are input into the target posture estimation model, the target posture estimation model can output the posture label expected at the next moment.

[0089] S230, collecting the images to be processed corresponding to the target subject at the current moment and the previous moment.

[0090] S240: Process each image to be processed based on the target posture estimation model to obtain posture information of the target subject at the next moment.

[0091] The technical solution of this embodiment determines a training sample set based on a spliced ​​image formed by splicing the current video frame and the previous video frame and a posture label corresponding to the next video frame, and obtains a target posture estimation model based on the training sample set, so that when the images to be processed corresponding to the target subject at the current moment and the previous moment are collected, each image to be processed is processed based on the target posture estimation model to obtain the posture information of the target subject at the next moment, thereby achieving the technical effect of improving the accuracy and convenience of posture estimation.

[0092] Embodiment 3

[0093] As an alternative embodiment of the above embodiment, Figure 6 FIG. 1 is a flow chart of an image processing method provided according to Embodiment 3 of the present invention. For details, please refer to the following specific contents.

[0094] See also Figure 6 , a spliced ​​image can be obtained by cascading the image at time t and the image at time t+1 in the video / image sequence. For example, the image at time t is a 3D image of 1241*376*3, the image at time t+1 is a 3D image of 1241*376*3, and the spliced ​​image after cascading and cutting is a 6D image of 1404*508*6 as the spliced ​​image. You can also use a clipped video or monocular RGB image sequence as a dataset. Let the dataset X = {x1, x2, ..., x N}, the dataset contains N RGB images, i.e. video frames. The average RGB value of all video frames in dataset X, i.e. the grayscale mean is At each time t, subtract the grayscale mean from the RGB value of the current video frame The target grayscale value of the current video frame can be obtained, that is, Furthermore, the size of each video frame can be resized to 1404*508 to adapt each video frame to the U-Net network structure, and two consecutive RGB video frames can be cascaded to form a new image tensor, namely a spliced ​​image. The spliced ​​image can be input into the U-Net network, namely the feature extraction network, which is a network structure including downsampling and upsampling processes. The downsampling process is used to gradually display geometric (environmental) information. The continuous deepening of convolution will cause the loss of a lot of important information in the image, while the upsampling process strengthens the network's extraction of effective features by combining the geometric information of each layer in the downsampling and the input information in the upsampling, and gradually restores the accuracy of the image. In order to extract an effective geometric feature representation of the visual odometry problem, the spliced ​​image can be used as the input of the U-Net network, and then the U-Net network performs the automatic feature extraction operation to obtain the target feature map. U-Net uses RGB images instead of pre-processed corresponding images such as optical flow or depth images as input, which is conducive to the network learning an effective feature representation for the visual odometry problem. This feature representation not only compresses the original RGB image into a compact description, but also enhances the training process of continuous sequences. Therefore, the feature map output by U-Net is passed to the subsequent network for sequential modeling.

[0095] On the basis of the above scheme, in order to discover and exploit the correlation between image sequences taken on a long trajectory, the target feature map can be input into the RNN, which includes three stacked LSTM layers. Figure 4 , where the hidden state of one LSTM is the input of another LSTM, and each LSTM layer contains 800 hidden states. Each LSTM is associated with a time step. Given the input x at time t t , hidden state h t-1 and the memory unit m of the previous LSTM t-1 , the update rule of LSTM at the current time t is as follows:

[0096] i t =σ(W xi x t +W hi h t-1 +b i )

[0097] f t =σ(W xf x t +W hf h t-1 +b f )

[0098] g t =tanh(W xgx t +W hg h t-1 +b g )

[0099] m t =f t ⊙m t-1 +i t ⊙g t

[0100] o t =σ(W xo x t +W ho h t-1 +b o )

[0101] h t =o t ⊙tanh(m t )

[0102] Among them, ⊙ represents the element-wise product of two vectors; σ represents the sigmoid nonlinear function; tanh is the hyperbolic tangent nonlinear function; W represents the corresponding weight matrix; b represents the bias vector; i t 、f t , g t 、m t and t They correspond to the input gate, forget gate, input modulation gate, memory unit and output gate at time t respectively. The hidden state of LSTM1 can be used as the input of LSTM2, and the hidden state of LSTM2 can be used as the input of LSTM3. Through the processing of LSTM3, the posture data to be used is obtained.

[0103] Based on the above scheme, the posture attribute determination model to be trained can be trained based on the error value between the posture data to be used and the posture label of the image at time t+2. For example, a set of monocular RGB image sequences X ending at time t can be given. t ={x1,x2,...x t}, the visual odometer system based on U-Net and RNN calculates the conditional probability Y of the posture t ={y1,y2,...,y t}as follows:

[0104] p(Y t |X t )=p(y1,…,y t |x1,…,x t )

[0105] In order to maximize the conditional probability, a parameter θ is introduced so that θ *=argmaxp(Y t |X t ; θ), in order to learn the hyperparameter θ, this technical solution minimizes the posture label at time t Pose information estimated by the model The difference between them is the optimization target, and the displacement at time t is obtained. The loss function is shown in formula (1):

[0106]

[0107] Get the rotation angle (also called Euler angle) at time t The loss function is shown in formula (2):

[0108]

[0109] The final loss function of pose estimation is shown in formula (3):

[0110]

[0111] Among them, α is a scaling factor used to balance the weights of displacement and rotation angle losses; N is the number of samples. The loss function of the final model is shown in formula (4):

[0112]

[0113] The training error of the loss function (Formula (4)), that is, the loss parameter, can be used as a condition for detecting whether the loss function has reached convergence. The convergence of the loss function in the posture attribute determination model to be trained is used as the training goal to obtain the target posture attribute determination model.

[0114] On the basis of the above scheme, the effective geometric feature representation of the monocular visual odometer is automatically extracted by inputting the cascaded RGB image tensor into the U-Net network structure, and the visual odometer positioning method based on deep learning is extended to a new environment; at the same time, the context information of the sequence frame is combined, and the RNN network is used to automatically model the sequential dependency and complex motion dynamics of the image sequence to achieve accurate and robust positioning results. This technical solution adopts the technology of combining the U-Net network with the RNN network to estimate the posture directly from the original RGB image in an end-to-end manner, which not only improves the accuracy of posture estimation, but also improves the efficiency of posture estimation.

[0115] The technical solution of this embodiment is to collect the images to be processed corresponding to the target subject at the current moment and the previous moment, and process each image to be processed based on the target posture estimation model to obtain the posture information of the target subject at the next moment; wherein the target posture estimation model includes a feature extraction model and a target posture attribute determination model, and the target posture estimation model is determined based on a pre-constructed training sample set, and the training sample set includes multiple training samples, and the training samples include a spliced ​​image and a posture label. The spliced ​​image is based on the current video frame and the previous video frame, and the posture label corresponds to the next video frame. This solves the problem of low posture estimation accuracy caused by extracting geometric constraints from a single frame image based on a geometric algorithm in the prior art, and realizes that the target posture estimation model is pre-constructed by using the spliced ​​image based on the current video frame and the previous video frame and the posture label corresponding to the next video frame. When the images to be processed corresponding to the target subject at the current moment and the previous moment are collected, each image to be processed is processed based on the target posture estimation model to obtain the posture information of the target subject at the next moment, thereby achieving the technical effect of improving the accuracy and convenience of posture estimation.

[0116] Embodiment 4

[0117] Figure 7 is a schematic diagram of the structure of an image processing device provided according to Embodiment 4 of the present invention. Figure 7 As shown, the device includes: a to-be-processed image acquisition module 710 and a posture information determination module 720.

[0118] Among them, the image acquisition module 710 to be processed is used to acquire the images to be processed corresponding to the target subject at the current moment and the previous moment; the posture information determination module 720 is used to process each image to be processed based on the target posture estimation model to obtain the posture information of the target subject at the next moment; wherein the target posture estimation model includes a feature extraction model and a target posture attribute determination model, and the target posture estimation model is determined based on a pre-constructed training sample set, and the training sample set includes multiple training samples, and the training samples include a spliced ​​image and a posture label, and the spliced ​​image is based on the splicing of the current video frame and the previous video frame, and the posture label corresponds to the next video frame of the current video frame.

[0119] The technical solution of this embodiment is to collect the images to be processed corresponding to the target subject at the current moment and the previous moment, and process each image to be processed based on the target posture estimation model to obtain the posture information of the target subject at the next moment; wherein the target posture estimation model includes a feature extraction model and a target posture attribute determination model, and the target posture estimation model is determined based on a pre-constructed training sample set, and the training sample set includes multiple training samples, and the training samples include a spliced ​​image and a posture label. The spliced ​​image is based on the current video frame and the previous video frame, and the posture label corresponds to the next video frame. This solves the problem of low posture estimation accuracy caused by extracting geometric constraints from a single frame image based on a geometric algorithm in the prior art, and realizes that the target posture estimation model is pre-constructed by using the spliced ​​image based on the current video frame and the previous video frame and the posture label corresponding to the next video frame. When the images to be processed corresponding to the target subject at the current moment and the previous moment are collected, each image to be processed is processed based on the target posture estimation model to obtain the posture information of the target subject at the next moment, thereby achieving the technical effect of improving the accuracy and convenience of posture estimation.

[0120] On the basis of the above device, optionally, the device further includes a target posture estimation model determination module, including a training sample determination unit and a target posture estimation model determination unit.

[0121] A training sample determination unit, used to determine each training sample in the training sample set;

[0122] The target posture estimation model determination unit is used to train the posture estimation model to be trained based on each training sample to obtain the target posture estimation model, so as to process the to-be-processed images of the target subject at the current moment and the previous moment based on the target posture estimation model to obtain the posture information of the target subject at the next moment.

[0123] On the basis of the above device, optionally, the training sample determination unit includes a grayscale value to be used determination subunit, a grayscale mean determination subunit, a target grayscale value determination subunit and a training sample determination subunit.

[0124] A grayscale value determination subunit to be used, used to collect a plurality of continuous video frames and determine the grayscale value to be used corresponding to each video frame;

[0125] A grayscale mean value determination subunit, used for determining a grayscale mean value based on each grayscale value to be used;

[0126] a target grayscale value determination subunit, configured to determine, for each video frame, a target grayscale value of the current video frame based on the grayscale mean value and the grayscale value to be used of the current video frame;

[0127] The training sample determination subunit is used to determine each training sample based on the target grayscale value and posture label corresponding to each video frame.

[0128] On the basis of the above device, optionally, the training sample determination subunit includes a splicing image determination subunit and a training sample determination subunit.

[0129] A spliced ​​image determination subunit is used to determine, for each video frame, a spliced ​​image corresponding to the current video frame based on target grayscale values ​​corresponding to the current video frame and the previous video frame;

[0130] The training sample determination subunit is used to determine the training sample based on the spliced ​​image corresponding to the corresponding video frame and the posture label of the next video frame corresponding to the video frame.

[0131] On the basis of the above device, optionally, the target posture estimation model determination unit includes a posture attribute determination model training subunit to be trained, a target posture attribute determination model determination subunit and a target posture estimation model determination subunit.

[0132] The training subunit of the posture attribute determination model to be trained is used for processing the spliced ​​image in the current training sample based on the feature extraction model for each training sample to obtain a target feature map corresponding to the spliced ​​image, and taking the target feature map as the input of the posture attribute determination model to be trained, taking the corresponding posture label as the output of the posture attribute determination model to be trained, and training the posture attribute determination model to be trained;

[0133] A target posture attribute determination model determination subunit is used to take the convergence of the loss function in the posture attribute determination model to be trained as a training target to obtain a target posture attribute determination model;

[0134] The target posture estimation model determination subunit is used to splice the feature extraction model and the target posture attribute determination model to obtain the target posture estimation model.

[0135] On the basis of the above-mentioned device, optionally, the posture attribute to be trained determines the model training subunit, including a first feature map to be used determining small unit, a second feature map to be used determining small unit, a third feature map to be used determining small unit and a target feature map determining small unit.

[0136] A first feature map to be used determining unit is used to process the spliced ​​image based on a first preset number of convolutional layers according to a first sampling rate to obtain a first feature map to be processed, and downsample the first feature map to be processed based on a second sampling rate to obtain a first feature map to be used;

[0137] a second feature map to be used determining unit, configured to process the first feature map to be used according to the first sampling rate based on a second preset number of convolutional layers to obtain a second feature map to be processed, and downsample the second feature map to be processed based on the second sampling rate to obtain a second feature map to be used;

[0138] a third feature map to be used determining unit, configured to process the second feature map to be used according to the first sampling rate based on a third preset number of convolutional layers to obtain a third feature map to be processed, and downsample the third feature map to be processed based on the second sampling rate to obtain a third feature map to be used;

[0139] The target feature map determines a small unit, which is used to process the third feature map to be used according to the first sampling rate based on a fourth preset number of convolutional layers to obtain a fourth feature map to be processed, and downsample the fourth feature map to be processed based on the second sampling rate to obtain a fourth feature map to be used, and upsample the fourth feature map to be used to obtain the target feature map.

[0140] On the basis of the above-mentioned device, optionally, the target feature map determines a small unit, including a fourth feature map to be convolved determining micro unit, a third feature map to be convolved determining micro unit, a second feature map to be convolved determining micro unit, a first feature map to be spliced ​​determining micro unit and a target feature map determining micro unit.

[0141] The fourth feature map to be convolved determines a micro unit, which is used to upsample the fourth feature map to be used based on the third sampling rate to obtain a fourth feature map to be spliced, and splice the fourth feature map to be spliced ​​with the sheared fourth feature map to be processed to obtain a fourth feature map to be convolved;

[0142] The third feature map to be convolved determines a micro unit, which is used to process the fourth feature map to be convolved according to the first sampling rate based on the fourth preset number of convolutional layers to obtain a fourth feature map to be sampled, and upsample the fourth feature map to be sampled based on the third sampling rate to obtain a third feature map to be spliced, and splice the third feature map to be spliced ​​with the sheared third feature map to be processed to obtain a third feature map to be convolved;

[0143] The second feature map to be convolved determines a micro unit, which is used to process the third feature map to be convolved according to the first sampling rate based on the third preset number of convolution layers to obtain a third feature map to be sampled, and upsample the third feature map to be sampled based on the third sampling rate to obtain a second feature map to be spliced, and splice the second feature map to be spliced ​​with the sheared second feature map to be processed to obtain a second feature map to be convolved;

[0144] The first feature map to be spliced ​​determines a micro unit, which is used to process the second feature map to be convolved according to the first sampling rate based on the second preset number of convolutional layers to obtain a second feature map to be sampled, and upsample the second feature map to be sampled based on the third sampling rate to obtain a first feature map to be spliced;

[0145] The target feature map determines a micro unit, which is used to splice the first feature map to be spliced ​​with the cut first feature map to be processed to obtain a first feature map to be convolved, and processes the first feature map to be convolved according to the first sampling rate based on the first preset number of convolution layers to obtain a target feature map.

[0146] On the basis of the above device, optionally, the posture attribute to be trained model determination training subunit includes a posture attribute to be trained model determination subunit, a posture data to be used determination subunit and a posture attribute to be trained model determination training subunit.

[0147] A small unit for determining a posture attribute determination model to be trained, used for cascading a plurality of posture memory models to be used to construct the posture attribute determination model to be trained;

[0148] A small unit for determining posture data to be used, used for processing the target feature map based on each posture memory model to be used to obtain posture data to be used;

[0149] The posture attribute determination model training subunit to be trained is used to train the posture attribute determination model to be trained based on the error value between the posture data to be used and the posture label.

[0150] The image processing device provided by the embodiment of the present invention can execute the image processing method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0151] Embodiment 5

[0152] Figure 8 : is a schematic diagram of the structure of an electronic device that implements the image processing method of an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0153] like Figure 8As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0154] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0155] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as an image processing method.

[0156] In some embodiments, the image processing method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the image processing method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the image processing method in any other appropriate manner (e.g., by means of firmware).

[0157] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0158] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0159] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0160] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0161] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0162] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0163] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0164] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An image processing method, characterized in that: include: Collecting the images to be processed corresponding to the target subject at the current moment and the previous moment; wherein the target subject is a vehicle; Processing each image to be processed based on the target posture estimation model to obtain the posture information of the target subject at the next moment; wherein the posture information includes displacement and rotation angle; The target posture estimation model includes a feature extraction model and a target posture attribute determination model, the target posture estimation model is determined based on a pre-constructed training sample set, the training sample set includes multiple training samples, the training samples include a spliced ​​image and a posture label, the spliced ​​image is based on the current video frame and the previous video frame, and the posture label corresponds to the next video frame of the current video frame; Collect multiple continuous video frames and determine the grayscale value to be used corresponding to each video frame; Based on each gray value to be used, determining a gray mean; For each video frame, based on the grayscale mean value and the grayscale value to be used of the current video frame, determine a target grayscale value of the current video frame; Determine each training sample in the training sample set based on the target grayscale value and posture label corresponding to each video frame; The posture estimation model to be trained is trained based on the training samples to obtain the target posture estimation model, and the images to be processed of the target subject at the current moment and the previous moment are processed based on the target posture estimation model to obtain the posture information of the target subject at the next moment.

2. The method according to claim 1, characterized in that The step of determining each training sample in the training sample set based on the target grayscale value and the posture label corresponding to each video frame includes: For each video frame, based on the target grayscale values ​​corresponding to the current video frame and the previous video frame, determine a spliced ​​image corresponding to the current video frame; A training sample is determined based on a stitched image corresponding to a corresponding video frame and a pose label of a next video frame corresponding to the video frame.

3. The method according to claim 1, characterized in that The training of the to-be-trained posture estimation model based on the training samples to obtain the target posture estimation model includes: For each training sample, the spliced ​​image in the current training sample is processed based on the feature extraction model to obtain a target feature map corresponding to the spliced ​​image, and the target feature map is used as the input of the posture attribute determination model to be trained, and the corresponding posture label is used as the output of the posture attribute determination model to be trained, so as to train the posture attribute determination model to be trained; Taking the convergence of the loss function in the posture attribute determination model to be trained as a training target, and obtaining a target posture attribute determination model; The feature extraction model and the target posture attribute determination model are spliced ​​to obtain the target posture estimation model.

4. The method according to claim 3, characterized in that The step of processing the spliced ​​image in the current training sample based on the feature extraction model to obtain a target feature map corresponding to the spliced ​​image includes: Processing the stitched image based on a first preset number of convolutional layers at a first sampling rate to obtain a first feature map to be processed, and downsampling the first feature map to be processed based on a second sampling rate to obtain a first feature map to be used; Processing the first feature map to be used based on a second preset number of convolutional layers according to the first sampling rate to obtain a second feature map to be processed, and downsampling the second feature map to be processed based on the second sampling rate to obtain a second feature map to be used; Processing the second feature map to be used according to the first sampling rate based on a third preset number of convolutional layers to obtain a third feature map to be processed, and downsampling the third feature map to be processed based on the second sampling rate to obtain a third feature map to be used; The third feature map to be used is processed according to the first sampling rate based on a fourth preset number of convolutional layers to obtain a fourth feature map to be processed, and the fourth feature map to be processed is downsampled based on the second sampling rate to obtain a fourth feature map to be used, and the fourth feature map to be used is upsampled to obtain the target feature map.

5. The method according to claim 4, characterized in that The up-sampling of the fourth feature map to be used to obtain the target feature map includes: Upsampling the fourth feature map to be used based on the third sampling rate to obtain a fourth feature map to be spliced, and splicing the fourth feature map to be spliced ​​with the sheared fourth feature map to be processed to obtain a fourth feature map to be convolved; Processing the fourth feature map to be convolved according to the first sampling rate based on the fourth preset number of convolutional layers to obtain a fourth feature map to be sampled, upsampling the fourth feature map to be sampled based on the third sampling rate to obtain a third feature map to be spliced, and splicing the third feature map to be spliced ​​with the sheared third feature map to be processed to obtain a third feature map to be convolved; Processing the third feature map to be convolved based on the third preset number of convolutional layers according to the first sampling rate to obtain a third feature map to be sampled, upsampling the third feature map to be sampled based on the third sampling rate to obtain a second feature map to be spliced, and splicing the second feature map to be spliced ​​with the sheared second feature map to be processed to obtain a second feature map to be convolved; Processing the second feature map to be convolved according to the first sampling rate based on the second preset number of convolutional layers to obtain a second feature map to be sampled, and upsampling the second feature map to be sampled based on the third sampling rate to obtain a first feature map to be spliced; The first feature map to be spliced ​​is spliced ​​with the cut first feature map to be processed to obtain a first feature map to be convolved, and the first feature map to be convolved is processed according to the first sampling rate based on the first preset number of convolutional layers to obtain a target feature map.

6. The method according to claim 3, characterized in that: The method of using the target feature map as the input of the posture attribute determination model to be trained, using the corresponding posture label as the output of the posture attribute determination model to be trained, and training the posture attribute determination model to be trained includes: Cascading multiple posture memory models to be used to construct the posture attribute determination model to be trained; Processing the target feature map based on each posture memory model to be used to obtain posture data to be used; The posture attribute determination model to be trained is trained based on the error value between the posture data to be used and the posture label.

7. An image processing device, characterized in that: include: The to-be-processed image acquisition module is used to acquire the to-be-processed images corresponding to the target subject at the current moment and the previous moment; wherein the target subject is a vehicle; A posture information determination module, used to process each to-be-processed image based on a target posture estimation model to obtain the posture information of the target subject at the next moment; wherein the posture information includes displacement and rotation angle; The target posture estimation model includes a feature extraction model and a target posture attribute determination model, the target posture estimation model is determined based on a pre-constructed training sample set, the training sample set includes multiple training samples, the training samples include a spliced ​​image and a posture label, the spliced ​​image is based on the current video frame and the previous video frame, and the posture label corresponds to the next video frame of the current video frame; A target posture estimation model determination module includes: a training sample determination unit and a target posture estimation model determination unit; The training sample determination unit includes: a grayscale value to be used determination subunit, a grayscale mean determination subunit, a target grayscale value determination subunit and a training sample determination subunit; The grayscale value to be used determining subunit is used to collect a plurality of continuous video frames and determine the grayscale value to be used corresponding to each video frame; The grayscale mean value determining subunit is used to determine the grayscale mean value based on each grayscale value to be used; The target grayscale value determination subunit is used to determine, for each video frame, a target grayscale value of the current video frame based on the grayscale mean value and the grayscale value to be used of the current video frame; The training sample determination subunit is used to determine each training sample in the training sample set based on the target grayscale value and posture label corresponding to each video frame; The target posture estimation model determination unit is used to train the posture estimation model to be trained based on the training samples to obtain the target posture estimation model, and to process the images to be processed of the target subject at the current moment and the previous moment based on the target posture estimation model to obtain the posture information of the target subject at the next moment.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the image processing method according to any one of claims 1 to 6 when executed.

Images