Method, device and electronic device for adjusting depth image prediction model

Abstract

The embodiments of the present invention disclose a method, device, readable storage medium and electronic device for adjusting a depth image prediction model. The embodiments of the present invention obtain a sequence of sequentially shot images; obtain multiple sample data groups, each sample data group includes a first image and a second image adjacent to each other in the image sequence, and motion data recorded in a time period between the shooting of the first image and the second image; and iteratively adjust the parameters of the depth image prediction model and the motion prediction model in an unsupervised manner to optimize the loss function. Through the above method, the parameters of the depth image prediction model and the motion prediction model are iteratively adjusted in an unsupervised manner to optimize the loss function, and the optimized depth image prediction model and motion prediction model are obtained. The depth image is determined by the optimized depth image prediction model, which can improve the accuracy of the estimated depth image.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a method, device and electronic device for adjusting a depth image prediction model. Background Art

[0002] In recent years, the research on autonomous driving technology has become more and more in-depth and extensive. Autonomous driving cars are an important application of autonomous driving technology. Autonomous driving cars are a type of smart car. Vehicles running in autonomous driving mode can free passengers, especially drivers, from some driving-related responsibilities. When running in autonomous driving mode, the vehicle mainly relies on the intelligent driving instrument based on the computer system in the vehicle to achieve the purpose of autonomous driving, thereby allowing the vehicle to drive without human-machine interaction. A specific technology in autonomous driving technology is to obtain a depth image through a monocular image and then predict the depth of the scene. The depth image, also known as the range image, refers to an image that uses the distance from the image collector to each point in the scene as a pixel value, that is, each pixel value in the depth image represents the distance between a point in the scene and the camera.

[0003] In the prior art, due to the influence of lighting on monocular images, the influence of rapid movement of the camera taking the monocular image, and scale blur in the monocular image, the accuracy of the depth image obtained by the monocular image is poor. How to improve the accuracy of the depth image generated by the monocular image is a problem that needs to be solved at present. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a method, device, readable storage medium and electronic device for adjusting a depth image prediction model, which iteratively adjusts the parameters of the depth image prediction model and the motion prediction model in an unsupervised manner to optimize the loss function, thereby improving the accuracy of the depth image generated by the depth image prediction model.

[0005] In a first aspect, an embodiment of the present invention provides a method for adjusting a depth image prediction model, the method comprising: acquiring a sequence of images taken sequentially; acquiring multiple sample data groups, each sample data group comprising a first image and a second image adjacent to each other in the image sequence, and motion data recorded in a time period between taking the first image and the second image; iteratively adjusting parameters of a depth image prediction model and a motion prediction model in an unsupervised manner to optimize a loss function; wherein the step of determining the loss function comprises: determining a motion matrix according to the sample data group based on the motion prediction model; predicting a first initial depth image corresponding to the first image and a second initial depth image corresponding to the second image based on the depth image prediction model; determining the loss function according to the first initial depth image, the second initial depth image and the motion matrix.

[0006] Preferably, the method further comprises: inputting at least one third image into the iteratively adjusted depth image prediction model, and outputting a third initial depth image corresponding to the third image.

[0007] Preferably, the adjacent third image and fourth image and the motion data recorded in the time period between shooting the third image and the fourth image are input into the iteratively adjusted motion prediction model, and the motion matrix is ​​output, wherein the motion matrix is ​​used to determine the motion trajectory of the device shooting the third image and the fourth image.

[0008] Preferably, the motion data includes inertial measurement unit IMU data; the motion prediction model includes a first motion encoder PoseNet, a second motion encoder BiLSTM, and a gated weighted network; and the depth image prediction model includes an image encoder DepthNet.

[0009] Preferably, determining the motion matrix according to the sample data group specifically includes:

[0010] determining a first motion feature according to the adjacent first image and the second image;

[0011] determining a second motion feature based on the IMU data;

[0012] The first motion feature and the second motion feature are selectively fused to determine the motion matrix.

[0013] Preferably, determining the first motion feature according to the adjacent first image and the second image specifically includes:

[0014] stacking the adjacent first image and second image;

[0015] Extracting first motion features of the adjacent first and second images of the stack according to the first motion encoder PoseNet.

[0016] Preferably, determining the second motion feature according to the IMU data specifically includes:

[0017] The second motion feature corresponding to the IMU data is determined by the second motion encoder BiLSTM.

[0018] Preferably, the selectively fusing the first motion feature with the second motion feature to determine the motion matrix specifically includes:

[0019] Determining a first weight coefficient corresponding to the first motion feature according to a gated weighted network, and determining a second weight coefficient corresponding to the second motion feature according to the gated weighted network;

[0020] The motion matrix is ​​determined according to the first motion feature, the first weight coefficient, the second motion feature, and the second weight coefficient.

[0021] Preferably, determining a first weight coefficient corresponding to the first motion feature according to a gated weighted network specifically includes:

[0022] The first motion feature is passed through the fully connected network to determine a first intermediate parameter;

[0023] A first weight coefficient corresponding to the first motion feature is determined according to the first intermediate parameter and the Sigmoid function.

[0024] Preferably, determining the second weight coefficient corresponding to the second motion feature according to a gated weighted network specifically includes:

[0025] The second motion feature is passed through the fully connected network to determine a second intermediate parameter;

[0026] A second weight coefficient corresponding to the second motion feature is determined according to the second intermediate parameter and the Sigmoid function.

[0027] Preferably, the IMU data includes three-dimensional acceleration data and three-dimensional gyroscope data.

[0028] Preferably, the method further comprises:

[0029] Determine, according to the two-dimensional coordinates of any pixel point in the first image, the three-dimensional coordinates of the any pixel point in the first initial depth image corresponding to the first image;

[0030] According to the three-dimensional coordinates and the motion matrix, a first parameter of reprojecting the first image to the adjacent second image and a second parameter of reprojecting the first depth image corresponding to the first image to the adjacent second depth image corresponding to the second image are determined.

[0031] Preferably, determining the loss function according to the first initial depth image, the second initial depth image and the motion matrix specifically includes:

[0032] The loss function is determined according to the first parameter and the second parameter.

[0033] Preferably, the loss function includes an image reconstruction loss function, a depth consistency loss function and a depth smoothness loss function.

[0034] In a second aspect, an embodiment of the present invention provides a device for adjusting a depth image prediction model, the device comprising:

[0035] An acquisition unit is used to acquire a sequence of images taken sequentially; the acquisition unit is also used to acquire multiple sample data groups, each sample data group includes a first image and a second image adjacent to each other in the image sequence, and motion data recorded in a time period between the shooting of the first image and the second image; an adjustment unit is used to iteratively adjust the parameters of a depth image prediction model and a motion prediction model in an unsupervised manner so as to optimize a loss function; wherein the step of determining the loss function is processed by a determination unit, and the determination unit is specifically used to: determine a motion matrix according to the sample data group based on the motion prediction model; predict a first initial depth image corresponding to the first image and a second initial depth image corresponding to the second image based on the depth image prediction model; and determine the loss function according to the first initial depth image, the second initial depth image and the motion matrix.

[0036] Preferably, the device further comprises: a first processing unit, configured to input at least one third image into the iteratively adjusted depth image prediction model, and output a third initial depth image corresponding to the third image.

[0037] Preferably, the device also includes: a second processing unit, used for inputting adjacent third and fourth images and motion data recorded in the time period between shooting the third and fourth images into the iteratively adjusted motion prediction model, and outputting the motion matrix, wherein the motion matrix is ​​used to determine the motion trajectory of the device shooting the third and fourth images.

[0038] Preferably, the motion data includes inertial measurement unit IMU data; the motion prediction model includes a first motion encoder PoseNet, a second motion encoder BiLSTM, and a gated weighted network; and the depth image prediction model includes an image encoder DepthNet.

[0039] Preferably, the determining unit is specifically used for:

[0040] determining a first motion feature according to the adjacent first image and the second image;

[0041] determining a second motion feature based on the IMU data;

[0042] The first motion feature and the second motion feature are selectively fused to determine the motion matrix.

[0043] Preferably, the determined unit is further used for:

[0044] stacking the adjacent first image and second image;

[0045] Extracting first motion features of the adjacent first and second images of the stack according to the first motion encoder PoseNet.

[0046] Preferably, the determined unit is further used for:

[0047] The second motion feature corresponding to the IMU data is determined by the second motion encoder BiLSTM.

[0048] Preferably, the determining unit is further configured to:

[0049] Determining a first weight coefficient corresponding to the first motion feature according to a gated weighted network, and determining a second weight coefficient corresponding to the second motion feature according to the gated weighted network;

[0050] The motion matrix is ​​determined according to the first motion feature, the first weight coefficient, the second motion feature, and the second weight coefficient.

[0051] Preferably, the determining unit is specifically used for:

[0052] The first motion feature is passed through the fully connected network to determine a first intermediate parameter;

[0053] A first weight coefficient corresponding to the first motion feature is determined according to the first intermediate parameter and the Sigmoid function.

[0054] Preferably, the determining unit is specifically used for:

[0055] The second motion feature is passed through the fully connected network to determine a second intermediate parameter;

[0056] A second weight coefficient corresponding to the second motion feature is determined according to the second intermediate parameter and the Sigmoid function.

[0057] Preferably, the IMU data includes three-dimensional acceleration data and three-dimensional gyroscope data.

[0058] Preferably, the determining unit is further used for:

[0059] Determine, according to the two-dimensional coordinates of any pixel point in the first image, the three-dimensional coordinates of the any pixel point in the first initial depth image corresponding to the first image;

[0060] According to the three-dimensional coordinates and the motion matrix, a first parameter of reprojecting the first image to the adjacent second image and a second parameter of reprojecting the first depth image corresponding to the first image to the adjacent second depth image corresponding to the second image are determined.

[0061] Preferably, the determining unit is specifically used for:

[0062] The loss function is determined according to the first parameter and the second parameter.

[0063] Preferably, the loss function includes an image reconstruction loss function, a depth consistency loss function and a depth smoothness loss function.

[0064] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement a method as described in the first aspect or any possible embodiment of the first aspect.

[0065] In a fourth aspect, an embodiment of the present invention provides an electronic device, comprising a memory and a processor, the memory being used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement a method as described in the first aspect or any possible embodiment of the first aspect.

[0066] The embodiment of the present invention obtains a sequence of sequentially shot images; obtains multiple sample data groups, each sample data group includes a first image and a second image adjacent to each other in the image sequence, and motion data recorded in the time period between the shooting of the first image and the second image; iteratively adjusts the parameters of the depth image prediction model and the motion prediction model in an unsupervised manner to optimize the loss function; wherein the step of determining the loss function includes: determining a motion matrix according to the sample data group based on the motion prediction model; predicting a first initial depth image corresponding to the first image and a second initial depth image corresponding to the second image based on the depth image prediction model; and determining the loss function according to the first initial depth image, the second initial depth image and the motion matrix. Through the above method, the parameters of the depth image prediction model and the motion prediction model are iteratively adjusted in an unsupervised manner to optimize the loss function, and the optimized depth image prediction model and motion prediction model are obtained. The depth image is determined by the optimized depth image prediction model, and the accuracy of the depth image can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0068] Figure 1 is a flow chart of a method for adjusting a depth image prediction model according to an embodiment of the present invention;

[0069] Figure 2 is a flow chart of a method for determining a loss function according to an embodiment of the present invention;

[0070] Figure 3 is a flow chart of a first weight coefficient generating method according to an embodiment of the present invention;

[0071] Figure 4 is a flow chart of a second weight coefficient generating method according to an embodiment of the present invention;

[0072] Figure 5 is a flow chart of a parameter generation method according to an embodiment of the present invention;

[0073] Figure 6 is a flow chart of a method for adjusting a depth image prediction model according to an embodiment of the present invention;

[0074] Figure 7 is a schematic diagram of a device for adjusting a depth image prediction model according to an embodiment of the present invention;

[0075] Figure 8 is a schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0076] The present invention is described below based on the embodiments, but the present invention is not limited to these embodiments. In the detailed description of the present invention below, some specific details are described in detail. It is possible for a person skilled in the art to fully understand the present invention without the description of these details. In order to avoid confusing the essence of the present invention, known methods, processes, procedures, components and circuits are not described in detail.

[0077] In addition, persons of ordinary skill in the art will appreciate that the drawings provided herein are for illustration purposes and are not necessarily drawn to scale.

[0078] Unless the context clearly requires otherwise, the words "include", "comprising" and similar words throughout the application should be interpreted as including rather than exclusive or exhaustive; that is, the meaning is "including but not limited to".

[0079] In the description of the present invention, it should be understood that the terms "first", "second", etc. are used for descriptive purposes only and cannot be understood as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise specified, the meaning of "plurality" is two or more.

[0080] An important technology in autonomous driving technology is to obtain a depth image through a monocular image and then predict the depth of the scene. The depth image refers to an image that uses the distance from the image collector (for example, a camera or a camera, etc.) to each point in the scene as a pixel value, that is, each pixel value in the depth image represents the distance between a certain point in the scene and the camera. Therefore, the depth image obtained through a monocular image plays an important role in the process of autonomous driving, and the accuracy of the depth image is also very important; in the prior art, in the process of obtaining a depth image through a monocular image, a depth estimation algorithm based on a monocular image is usually used. Specifically, it can be divided into two methods, method 1, a depth estimation algorithm based on depth clues, and method 2, a depth estimation algorithm based on machine learning; in method 1, commonly used depth clues include: motion information, linear perspective, focus, occlusion, texture, and shadow. Solving the depth based on motion information is the most commonly used method: the Structure from Motion (SfM) technology can recover the internal and external parameters of the camera and the depth information of the scene from the image sequence taken by the camera; the depth estimation algorithm based on the SfM technology can usually recover the scene depth information, but it is subject to special scenes and requires the movement of the camera; when there are moving objects in the scene, the accuracy of the depth solution is also greatly affected; because of the need to solve the internal and external parameters of the camera, the speed of depth estimation is also relatively slow; and when the monocular image is affected by lighting, it may be impossible to calculate the movement of the camera itself, and the rapid movement of the camera that takes the monocular image will cause the first frame image and the second frame image to be too different, which also has a great impact on the accuracy of the depth solution; in the second method, a large number of training images and corresponding depth images are input into the defined model for supervised learning through machine learning methods. After the model training is completed, the actual image to be tested can be input into the trained model for depth calculation. However, since there is no better depth image as input during the training process, it is impossible to train a model with higher accuracy. In summary, the accuracy of the depth image obtained by the monocular image in the prior art is poor. How to improve the accuracy of the depth image generated by the monocular image is a problem that needs to be solved at present.

[0081] In an embodiment of the present invention, a monocular image is combined with an inertial measurement unit (IMU) data, and a depth image prediction model and a motion prediction model are iteratively trained in an unsupervised manner. The addition of IMU data can overcome the problems of the monocular image being affected by illumination during the process of obtaining a depth image from a monocular image in the prior art, the influence of the rapid movement of the camera that takes the monocular image, and scale blur in the monocular image. Therefore, the depth image prediction model iteratively trained in an unsupervised manner has a higher accuracy.

[0082] In the embodiment of the present invention, Figure 1 1 is a flow chart of a method for adjusting a depth image prediction model of the present invention. Figure 1 As shown, the specific steps include:

[0083] Step S100: Acquire a sequence of sequentially shot images.

[0084] In a possible implementation, the sequentially shot image sequence is shot by the same image acquisition device (such as a camera) in chronological order, and the shooting time interval may be 0.1s, 1s, etc. The embodiment of the present invention does not limit this. The images are sorted according to the shooting time. Assuming that 10 images are shot in chronological order, the image sequence is specifically image 1, image 2, image 3, image 4, image 5, image 6, image 7, image 8, image 9 and image 10.

[0085] Step S101: Acquire multiple sample data groups, each sample data group including a first image and a second image adjacent to each other in an image sequence, and motion data recorded in a time period between capturing the first image and the second image.

[0086] In a possible implementation, assuming that the image sequence is the above-mentioned image 1, image 2, image 3, image 4, image 5, image 6, image 7, image 8, image 9 and image 10, any two adjacent images can be called the first image and the second image, for example, image 1 is the first image and image 2 is the second image; or image 2 is the first image and image 3 is the second image, and so on, the embodiments of the present invention are not repeated here; when image 1 is the first image and image 2 is the second image, the motion data recorded in the time period between image 1 and image 2 is determined, and the motion data includes inertial measurement unit IMU data, and the IMU data includes three-dimensional acceleration data and three-dimensional gyroscope data. The loss function can be determined by acquiring the first image, the second image, and the IMU data. The specific processing process is as follows: Figure 2 shown.

[0087] Step S102: iteratively adjust the parameters of the depth image prediction model and the motion prediction model in an unsupervised manner to optimize the loss function.

[0088] In one possible implementation, the first image, the second image, and the IMU data are used as inputs. Since the depth image prediction model and the motion prediction model are unlabeled models (i.e., no real values), the models cannot be trained using the commonly used known input and output methods. The loss function can be determined by obtaining the first image, the second image, and the IMU data, and the parameters of the depth image prediction model and the motion prediction model are iteratively adjusted in an unsupervised manner using the above inputs to determine the optimal parameters of the depth image prediction model and the motion prediction model so as to optimize the loss function. At this time, the depth image prediction model and the motion prediction model corresponding to the parameters of the optimal depth image prediction model and the motion prediction model are the depth image prediction model and the motion prediction model with higher accuracy that need to be obtained in the embodiments of the present invention.

[0089] In a possible implementation manner, after determining the iteratively adjusted depth image prediction model, the method further includes:

[0090] At least one third image is input into the iteratively adjusted depth image prediction model, and a third initial depth image corresponding to the third image is output.

[0091] In a possible implementation, since the parameters of the iteratively adjusted depth image prediction model optimize the loss function, the accuracy of the initial depth image determined according to the iteratively adjusted depth image prediction model is also high.

[0092] In a possible implementation, since the iteratively adjusted model includes not only the depth image prediction model but also the motion prediction model, after determining the iteratively adjusted depth image prediction model and the motion prediction model, the following is further included:

[0093] The adjacent third image and fourth image and the motion data recorded in the time period between shooting the third image and the fourth image are input into the iteratively adjusted motion prediction model, and the motion matrix is ​​output, wherein the motion matrix is ​​used to determine the motion trajectory of the device shooting the third image and the fourth image.

[0094] In a possible implementation, the depth image prediction model outputs a third initial depth image corresponding to the third image and a fourth initial depth image corresponding to the fourth image.

[0095] In an embodiment of the present invention, a sequence of sequentially shot images is obtained; a plurality of sample data groups are obtained, each sample data group includes a first image and a second image adjacent to each other in the image sequence, and motion data recorded in a time period between the shooting of the first image and the second image; the parameters of the depth image prediction model and the motion prediction model are iteratively adjusted in an unsupervised manner to optimize the loss function; wherein the step of determining the loss function includes: determining a motion matrix according to the sample data group based on the motion prediction model; predicting a first initial depth image corresponding to the first image and a second initial depth image corresponding to the second image based on the depth image prediction model; and determining the loss function according to the first initial depth image, the second initial depth image and the motion matrix. Through the above method, the parameters of the depth image prediction model and the motion prediction model are iteratively adjusted in an unsupervised manner to optimize the loss function, and the optimized depth image prediction model and the motion prediction model are obtained. The depth image is determined by the optimized depth image prediction model, so that the accuracy of the depth image can be improved; a more accurate device motion trajectory can be obtained by iteratively adjusting the motion prediction model.

[0096] In the embodiment of the present invention, Figure 2 is a flow chart of a method for determining the loss function of the present invention. Figure 2 As shown, the specific steps include:

[0097] Step S200: Determine a motion matrix according to the sample data group based on a motion prediction model.

[0098] In one possible implementation, the motion prediction model includes a first motion encoder, a second motion encoder, and a gated weighting network (Re weighting Network), wherein the first motion editor may be PoseNet, used to determine the first motion features of the adjacent first image and the second image, and the second motion encoder may be a bidirectional long short-term memory (Bi-directional Long Short-Term Memory, BiLSTM), used to determine the second motion features corresponding to the IMU data.

[0099] In a possible implementation, determining the motion matrix based on the sample data group specifically includes: determining a first motion feature based on the adjacent first image and second image; determining a second motion feature based on the IMU data; and performing selective feature fusion of the first motion feature and the second motion feature to determine the motion matrix.

[0100] In a possible implementation, determining a first motion feature based on the adjacent first image and the second image, wherein the first motion feature is a motion feature of a camera, specifically includes: stacking the adjacent first image and the second image; and extracting the first motion features of the stacked adjacent first image and the second image according to the first motion encoder PoseNet.

[0101] For example, it is assumed that the adjacent first image and the second image are two consecutive frames of images, namely, I and a and I b , the I a and I b After stacking together, the first motion characteristic is extracted by the first motion encoder PoseNet, and the PoseNet can be composed of 7 layers of two-dimensional convolutional neural networks with a step size of 2. The convolutional neural network is a feedforward neural network. The artificial neurons can respond to surrounding units and can perform large-scale image processing; in the two-dimensional convolutional neural network, the number of output channels of each layer is (16, 32, 64, 128, 256, 256, 256), and the convolution kernel sizes are (7, 5, 3, 3, 3, 3, 3), respectively. All layers use a linear rectified function (Rectified Linear Unit, ReLU) as an activation function; finally, the first motion feature determined by the PoseNet is obtained by f V Specifically:

[0102] f V =PoseNet(I a , I b )

[0103] In a possible implementation, determining the second motion feature based on the IMU data specifically includes: determining the second motion feature corresponding to the IMU data through the second motion encoder BiLSTM.

[0104] For example, the inertial measurement unit IMU can generate high-frequency, environmentally-independent and scale-consistent data, overcoming the shortcomings of monocular images. However, the IMU is easily affected by its own noise and bias, resulting in inestimable errors in the integration of the IMU data. Therefore, in the embodiment of the present invention, a bidirectional long short-term memory BiLSTM network containing 32 hidden layers is used as the motion editor of the IMU data. Since the BiLSTM network adopts a bidirectional design, it can capture the second motion features of the forward and backward order between two consecutive frames, corresponding to the forward and backward relationship when calculating the transformation between two consecutive frames. In the embodiment of the present invention, by represents the IMU data between two consecutive frames of images (the adjacent first image and the second image), then the second motion feature corresponding to the IMU data is expressed by f I Specifically:

[0105]

[0106] In a possible implementation, the first motion feature determined based on the image acquired by the camera is complementary to the second motion feature determined by the IMU data, that is, under appropriate lighting, the camera acquires accurate translation suitable for predicting static and different regions, while the IMU data provides high-frequency, consistent-scale motion estimation that is not affected by the environment. Specifically, assuming that the camera's shooting frequency is 0.1 seconds per frame, and the IMU data collection frequency is 0.01 seconds, therefore, the embodiment of the present invention selectively fuses the first motion feature with the second motion feature through a gated weighted network, and then determines a motion matrix suitable for different environmental conditions and object motion dynamics.

[0107] Specifically, the first motion feature and the second motion feature are selectively fused to determine the motion matrix, which specifically includes: determining a first weight coefficient corresponding to the first motion feature according to a Sigmoid function, and determining a second weight coefficient corresponding to the second motion feature according to the Sigmoid function; determining the motion matrix according to the first motion feature, the first weight coefficient, the second motion feature, and the second weight coefficient.

[0108] For example, the first weight coefficient is w V =Sigmoid V (f' V ), the second weight coefficient is w I =Sigmoid I (f' I ), the motion matrix is ​​P a→b =FC([w I *f I ;w V *f V ]), wherein the FC is used to represent the fully connected layer of the gated weighted network. Specifically, the first weight coefficient generation step is as follows: Figure 3 As shown, specifically including:

[0109] Step S300: The first motion feature is used to determine a first intermediate parameter through a fully connected network.

[0110] In the embodiment of the present invention, the intermediate parameter is f' I , f′ I =FC(f I), in fact, the f' I The first motion feature is generated after passing through several layers of fully connected layers, and the fully connected network is a part of the gated weighted network.

[0111] Step S301: Determine a first weight coefficient corresponding to the first motion feature according to the first intermediate parameter and a Sigmoid function.

[0112] Specifically, the Sigmoid function is part of the gated weighted network, and the nonlinear capability of the Sigmoid function is applied to form a gated operation to generate dynamic weights, wherein the first weight coefficient is w V =Sigmoid V (f' V ).

[0113] In a possible implementation, the second weight coefficient is w I =Sigmoid I (f' I ), specifically, the step of determining the second weight coefficient according to the Sigmoid function is as follows Figure 4 As shown, specifically including:

[0114] Step S400: The second motion feature is used to determine a second intermediate parameter through the fully connected network.

[0115] In the embodiment of the present invention, the intermediate parameter is f' V , f′ V =FC(f V ), in fact, the f' V The second motion feature is generated after passing through several layers of fully connected layers.

[0116] Step S401: Determine a second weight coefficient corresponding to the second motion feature according to the second intermediate parameter and the Sigmoid function.

[0117] Specifically, the nonlinear capability of the Sigmoid function is applied to form a gating operation to generate a dynamic weight, wherein the second weight coefficient is w I =Sigmoid I (f' I ).

[0118] Step S201: predicting a first initial depth image corresponding to the first image and a second initial depth image corresponding to the second image based on a depth image prediction model.

[0119] Specifically, the depth image prediction model includes an image encoder DepthNet. Optionally, the DepthNet can be a convolutional neural network (CNN). The depth image prediction model has not been iteratively adjusted, so the parameters in the depth image prediction model are the initial initialization parameters. Since the initialization parameters are used, the accuracy of the first initial depth image and the second initial depth image predicted by the depth image prediction model at this time is poor, and the parameters need to be adjusted subsequently according to the loss function.

[0120] Step S202: Determine the loss function according to the first initial depth image, the second initial depth image and the motion matrix.

[0121] In a possible implementation, the loss function includes an image reconstruction loss function, a depth consistency loss function, and a depth smoothness loss function.

[0122] Specifically, the loss function is expressed by L total Indicates that the image reconstruction loss function is expressed by L gc Indicates that the depth consistency loss function is obtained by L rec Indicates that the deep smoothing loss function is obtained by L ds Represented, the loss function is specifically:

[0123] L total =αL gc +βL rec +γL ds

[0124] Among them, α, β, and γ are parameters, and the specific values ​​of the parameters can be defined manually. For example, the values ​​are as follows: α=0.5, β=1, and γ=0.1.

[0125] The following is a detailed description of how to obtain the image reconstruction loss function, the depth consistency loss function, and the depth smoothing loss function:

[0126] The image reconstruction loss function is assumed to be a and I b Reconstructed with I a At the same pixel coordinates, the pixel values ​​remain consistent, and the structural similarity of the two images should also be maintained at a very high level. Therefore, the image reconstruction loss function is defined as L in this embodiment of the present invention. rec :

[0127]

[0128] Among them, the M recis the effective point mask calculated based on the predicted initial depth image and motion matrix. The effective point mask is the edge pixel point that does not appear in the camera. The image structure similarity SSIM is used to measure the structural similarity. 1 With λ 2 is a weight parameter, which can be set manually. For example, 1 is 0.15, the λ 2 The value is 0.85, which is not limited in the embodiment of the present invention.

[0129] In one possible implementation, how to a and I b Refactoring Production The specific process is as follows Figure 5 As shown: The details are as follows:

[0130] Step 500: Determine the three-dimensional coordinates of any pixel point in the first initial depth image corresponding to the first image according to the two-dimensional coordinates of the any pixel point in the first image.

[0131] For example, it is determined that the first image is I a , the I a The corresponding first initial depth image is D a , the first initial depth image is D a Each pixel in D a (i, j) indicates that, through geometric projection, the first initial depth image D a Each pixel in can be projected into a three-dimensional space point, which is represented by the following formula:

[0132] Q(i, j) = K -1 ·D a (i,j) · [i,j,1] T

[0133] Wherein, Q(i, j) represents the three-dimensional space point corresponding to the original pixel coordinates i, j after the initial depth image is projected, and K is the camera intrinsic parameter matrix.

[0134] Step 501: Determine first parameters for reprojecting the first image onto the adjacent second image according to the three-dimensional coordinates and the motion matrix.

[0135] Specifically, the motion matrix is ​​P a→b , reproject the 3D point cloud corresponding to the 3D coordinates to the adjacent frame, that is, a The pixel coordinates of each point are transformed to the next moment, and according to the transformed coordinates, the pixel coordinates of each point are transformed to the next moment, and ... bThe pixel values ​​corresponding to the above transformed coordinates are obtained by interpolation, and then generated

[0136] For example, the transformed coordinates are Said The specific formula is as follows:

[0137]

[0138] Said The specific formula is as follows:

[0139]

[0140] Among them, the first parameter is The image reconstruction loss function is determined according to the first parameter.

[0141] The depth consistency loss function, where the depth consistency means that the same real-world point will change in depth due to movement in two consecutive frames, and this change should be expressed by depth and motion. That is to say, the depth value predicted by a point at time t, after movement to time t+1, should be consistent with the depth value predicted by the point at time t+1; the loss formed by this is defined as the depth consistency loss L gc :

[0142] The depth consistency loss can also be called geometric consistency loss. gc It is used to represent the weight mask calculated by geometric consistency. Specifically, in the first depth map and the second depth map, if an object in the first depth map, such as a bicycle, moves in the second depth map, and the point that affects the calculation loss after the movement, the weight to exclude it is the weight mask. Indicates the depth value after the change, D′ b Indicates the depth value obtained after interpolation.

[0143] In the embodiment of the present invention, the This is the second parameter.

[0144] The depth smoothing loss function is used to regularize the depth estimation, which can optimize the depth with sharp changes and only allow the depth map with the original Figure 1 The specific formula is as follows:

[0145]

[0146] Among them, the It is used to represent the partial derivative of the X-axis. It is used to represent the partial derivative of the Y axis, where e is a natural constant.

[0147] The following is a detailed description of a method for adjusting the depth image prediction model through a complete flowchart. The flowchart can clearly understand the relationship between the depth image prediction model and the motion prediction model, as well as the processing of the image and IMU data, as shown in the figure. Figure 6 shown.

[0148] Specifically, the two images I a and I b Input into depthNet respectively to generate depth image D a and the depth image D b , image I a and I b Stacking is performed, and the first motion feature is generated through PoseNet; image I is obtained a and I b The IMU corresponding to the time period between the two images is 6-dimensional data. The IMU is input into the BiLSTM to generate a second motion feature. The first motion feature and the second motion feature are fused through a gated weighting network to generate a motion matrix. According to the motion matrix, the image I a and I b Processing and generation The depth image D a and D b Processing and generation

[0149] Figure 7 Schematic diagram of a device for training an image depth model according to an embodiment of the present invention. Figure 7 As shown, the apparatus of this embodiment includes an acquiring unit 701 , an adjusting unit 702 , and a determining unit 703 .

[0150] The acquisition unit 701 is used to acquire a sequence of images taken sequentially;

[0151] The acquisition unit 701 is further used to acquire a plurality of sample data groups, each sample data group including a first image and a second image adjacent to each other in an image sequence, and motion data recorded in a time period between taking the first image and the second image;

[0152] The adjustment unit 702 is used to iteratively adjust the parameters of the depth image prediction model and the motion prediction model in an unsupervised manner to optimize the loss function; wherein the step of determining the loss function is processed by the determination unit 703, and the determination unit 703 is specifically used to:

[0153] Based on the motion prediction model, determining a motion matrix according to the sample data group;

[0154] Based on the depth image prediction model, predicting a first initial depth image corresponding to the first image and a second initial depth image corresponding to the second image;

[0155] The loss function is determined according to the first initial depth image, the second initial depth image and the motion matrix.

[0156] Furthermore, the device also includes: a first processing unit, configured to input at least one third image into the iteratively adjusted depth image prediction model, and output a third initial depth image corresponding to the third image.

[0157] Furthermore, the device also includes: a second processing unit, used to input the adjacent third image and fourth image and the motion data recorded in the time period between shooting the third image and the fourth image into the iteratively adjusted motion prediction model, and output the motion matrix, wherein the motion matrix is ​​used to determine the motion trajectory of the device that shoots the third image and the fourth image.

[0158] Furthermore, the motion data includes inertial measurement unit IMU data; the motion prediction model includes the first motion encoder PoseNet, the second motion encoder BiLSTM, and the gated weighted network; and the depth image prediction model includes an image encoder DepthNnet.

[0159] Furthermore, the determination unit is specifically used to: determine a first motion feature based on the adjacent first image and second image; determine a second motion feature based on the IMU data; and perform selective feature fusion on the first motion feature and the second motion feature to determine the motion matrix.

[0160] Furthermore, the determination unit is specifically used to: stack the adjacent first image and the second image; and extract first motion features of the stacked adjacent first image and the second image according to the first motion encoder PoseNet.

[0161] Furthermore, the determination unit is specifically used to: determine the second motion feature corresponding to the IMU data through the second motion encoder BiLSTM.

[0162] Furthermore, the determination unit is specifically used to: determine a first weight coefficient corresponding to the first motion feature according to the gated weighted network, and determine a second weight coefficient corresponding to the second motion feature according to the gated weighted network; determine the motion matrix according to the first motion feature, the first weight coefficient, the second motion feature, and the second weight coefficient.

[0163] Furthermore, the determination unit is specifically used to: determine a first intermediate parameter of the first motion feature through a fully connected network; and determine a first weight coefficient corresponding to the first motion feature according to the first intermediate parameter and a Sigmoid function.

[0164] Furthermore, the determination unit is specifically used to: determine a second intermediate parameter of the second motion feature through the fully connected network; and determine a second weight coefficient corresponding to the second motion feature according to the second intermediate parameter and the Sigmoid function.

[0165] Furthermore, the IMU data includes three-dimensional acceleration data and three-dimensional gyroscope data.

[0166] Furthermore, the determining unit is further configured to:

[0167] According to the two-dimensional coordinates of any pixel point in the first image, determine the three-dimensional coordinates of the any pixel point in the first initial depth image corresponding to the first image; according to the three-dimensional coordinates and the motion matrix, determine the first parameter of reprojecting the first image to the adjacent second image, and the second parameter of reprojecting the first depth image corresponding to the first image to the adjacent second depth image corresponding to the second image.

[0168] Furthermore, the determination unit is specifically used to: determine the loss function according to the first parameter and the second parameter.

[0169] Furthermore, the loss function includes an image reconstruction loss function, a depth consistency loss function and a depth smoothness loss function.

[0170] Figure 8 Schematic diagram of an electronic device according to an embodiment of the present invention. Figure 8 As shown, Figure 8The electronic device shown is a device for adjusting a depth image prediction model, which includes a general computer hardware structure, which includes at least a processor 81 and a memory 82. The processor 81 and the memory 82 are connected via a bus 83. The memory 82 is suitable for storing instructions or programs executable by the processor 81. The processor 81 can be an independent microprocessor or a collection of one or more microprocessors. Thus, the processor 81 executes the instructions stored in the memory 82 to execute the method flow of the embodiment of the present invention as described above to realize the processing of data and the control of other devices. The bus 83 connects the above-mentioned multiple components together, and at the same time connects the above-mentioned components to the display controller 84 and the display device and the input / output (I / O) device 85. The input / output (I / O) device 85 can be a mouse, a keyboard, a modem, a network interface, a touch input device, a somatosensory input device, a printer, and other devices known in the art. Typically, the input / output device 85 is connected to the system via an input / output (I / O) controller 86.

[0171] Among them, the instructions stored in the memory 82 are executed by at least one processor 81 to achieve: obtaining a sequence of images taken sequentially; obtaining multiple sample data groups, each sample data group including a first image and a second image adjacent to each other in the image sequence, and motion data recorded in a time period between taking the first image and the second image; iteratively adjusting the parameters of the depth image prediction model and the motion prediction model in an unsupervised manner to optimize the loss function; wherein the step of determining the loss function includes: determining a motion matrix according to the sample data group based on the motion prediction model; predicting a first initial depth image corresponding to the first image and a second initial depth image corresponding to the second image based on the depth image prediction model; determining the loss function according to the first initial depth image, the second initial depth image and the motion matrix.

[0172] In an embodiment of the present invention, through the above method, the parameters of the depth image prediction model and the motion prediction model are iteratively adjusted in an unsupervised manner to optimize the loss function, and the optimized depth image prediction model and motion prediction model are obtained. The depth image is determined by the optimized depth image prediction model, which can improve the accuracy of the depth image; a more accurate device motion trajectory can be obtained by iteratively adjusting the motion prediction model.

[0173] In a possible implementation manner, the processor is further configured to:

[0174] At least one third image is input into the iteratively adjusted depth image prediction model, and a third initial depth image corresponding to the third image is output.

[0175] In the embodiment of the present invention, the third initial depth image determined by the optimized depth image prediction model has a higher accuracy.

[0176] In one possible implementation, the processor is further used to: input adjacent third and fourth images and motion data recorded in a time period between capturing the third and fourth images into the iteratively adjusted motion prediction model, and output the motion matrix, wherein the motion matrix is ​​used to determine the motion trajectory of the device capturing the third and fourth images.

[0177] In the embodiment of the present invention, the third initial depth image and the fourth initial depth image determined by the optimized depth image prediction model have high accuracy, and the motion matrix predicted by the optimized motion prediction model also has high accuracy.

[0178] In a possible implementation, the motion data includes inertial measurement unit IMU data; the motion prediction model includes a first motion encoder PoseNet, a second motion encoder BiLSTM, and a gated weighted network; and the depth image prediction model includes an image encoder DepthNnet.

[0179] In an embodiment of the present invention, the four models of the first motion encoder PoseNet, the second motion encoder BiLSTM, the gated weighted network and the image encoder DepthNnet are trained and adjusted together. After the training is completed, the image encoder DepthNnet can be used alone to determine the depth image. The image encoder DepthNnet can also be used together with the second motion encoder BiLSTM and the gated weighted network to determine the depth image and the motion matrix.

[0180] In a possible implementation, the processor is specifically used to: determine a first motion feature based on the adjacent first image and second image; determine a second motion feature based on the IMU data; and perform selective feature fusion of the first motion feature and the second motion feature to determine the motion matrix.

[0181] In the embodiment of the present invention, a better motion matrix can be generated by fusing the first motion feature with the second motion feature.

[0182] In a possible implementation, determining the first motion feature based on the adjacent first image and the second image specifically includes: stacking the adjacent first image and the second image; and extracting the first motion feature of the stacked adjacent first image and the second image according to the first motion encoder PoseNet.

[0183] In the embodiment of the present invention, the first motion features of the first image and the second image are extracted in the above manner, which may also be referred to as motion features of the visual part.

[0184] In a possible implementation, determining the second motion feature based on the IMU data specifically includes: determining the second motion feature corresponding to the IMU data through the second motion encoder BiLSTM.

[0185] In the embodiment of the present invention, the second motion feature corresponding to the IMU data is determined in the above manner, which may also be called the motion feature of the IMU.

[0186] In a possible implementation, the selective feature fusion of the first motion feature and the second motion feature to determine the motion matrix specifically includes: determining a first weight coefficient corresponding to the first motion feature according to a Sigmoid function, and determining a second weight coefficient corresponding to the second motion feature according to the Sigmoid function; determining the motion matrix according to the first motion feature, the first weight coefficient, the second motion feature, and the second weight coefficient.

[0187] In the embodiment of the present invention, a first weight coefficient applicable to the first motion feature and a second weight coefficient applicable to the second motion feature are determined by the gated weighted network, and then the performance of the motion matrix determined according to the two determined weight coefficients is better.

[0188] In a possible implementation, determining a first weight coefficient corresponding to the first motion feature according to the gated weighted network specifically includes: determining a first intermediate parameter of the first motion feature through a fully connected network; and determining a first weight coefficient corresponding to the first motion feature according to the first intermediate parameter and a Sigmoid function.

[0189] In a possible implementation, determining a second weight coefficient corresponding to the second motion feature according to the gated weighted network specifically includes: determining a second intermediate parameter of the second motion feature through the fully connected network; and determining a second weight coefficient corresponding to the second motion feature according to the second intermediate parameter and the Sigmoid function.

[0190] In a possible implementation, the IMU data includes three-dimensional acceleration data and three-dimensional gyroscope data.

[0191] In the embodiment of the present invention, the shortcomings of the first image and the second image can be overcome by using IMU data, which plays an important role in optimizing the depth image prediction model and the motion model.

[0192] In a possible implementation, the method also includes: determining the three-dimensional coordinates of any pixel point in the first initial depth image corresponding to the first image based on the two-dimensional coordinates of the any pixel point in the first image; determining the first parameters of reprojecting the first image to the adjacent second image, and the second parameters of reprojecting the first depth image corresponding to the first image to the adjacent second depth image corresponding to the second image based on the three-dimensional coordinates and the motion matrix.

[0193] In the embodiment of the present invention, the first parameter and the second parameter are indispensable factors in determining the loss function. Accurately determining the first parameter and the second parameter can be helpful in determining an accurate loss function.

[0194] In a possible implementation manner, determining the loss function according to the first initial depth image, the second initial depth image, and the motion matrix specifically includes: determining the loss function according to the first parameter and the second parameter.

[0195] In the embodiment of the present invention, the error of determining the loss function according to the first parameter and the second parameter is small.

[0196] In a possible implementation, the loss function includes an image reconstruction loss function, a depth consistency loss function, and a depth smoothness loss function.

[0197] In the embodiment of the present invention, the loss function constructed by using multiple types of loss functions has higher accuracy.

[0198] Specifically, the electronic device includes: one or more processors 81 and a memory 82, Figure 8 Take a processor 81 as an example. The processor 81 and the memory 82 may be connected via a bus or other means. Figure 8 In the example, the bus connection is used. The memory 82 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The processor 81 executes various functional applications and data processing of the device by running the non-volatile software programs, instructions and modules stored in the memory 82, that is, realizing the above-mentioned training image depth model method.

[0199] The memory 82 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store a list of options, etc. In addition, the memory 82 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 82 may optionally include a memory remotely arranged relative to the processor 81, and these remote memories may be connected to an external device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0200] One or more modules are stored in the memory 82, and when executed by one or more processors 81, the method for adjusting the depth image prediction model in any of the above method embodiments is executed.

[0201] The above-mentioned product can execute the method provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of the present application.

[0202] An embodiment of the present invention relates to a non-volatile storage medium for storing a computer-readable program, wherein the computer-readable program is used for a computer to execute part or all of the above method embodiments.

[0203] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including a number of instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0204] Those skilled in the art will appreciate that the above embodiments are specific embodiments for implementing the present invention, and that in actual applications, various changes may be made in form and detail without departing from the spirit and scope of the present invention.

Claims

1. A method for adjusting a depth image prediction model, characterized in that: The method includes: Acquire a sequence of sequentially captured images; Acquire a plurality of sample data groups, each sample data group comprising a first image and a second image adjacent to each other in an image sequence, and motion data recorded in a time period between taking the first image and the second image, wherein the motion data comprises inertial measurement unit (IMU) data; Iteratively adjusting the parameters of the depth image prediction model and the motion prediction model in an unsupervised manner to optimize the loss function; wherein the step of determining the loss function comprises: Based on the motion prediction model, determining a motion matrix according to the sample data group; Based on the depth image prediction model, predicting a first initial depth image corresponding to the first image and a second initial depth image corresponding to the second image; Determine the loss function according to the first initial depth image, the second initial depth image and the motion matrix; The step of determining the motion matrix according to the sample data group specifically includes: determining a first motion feature according to the adjacent first image and the second image; determining a second motion feature based on the IMU data; The first motion feature and the second motion feature are selectively fused to determine the motion matrix, wherein a first weight coefficient corresponding to the first motion feature is determined according to a gated weighted network, and a second weight coefficient corresponding to the second motion feature is determined according to the gated weighted network; the motion matrix is ​​determined according to the first motion feature, the first weight coefficient, the second motion feature, and the second weight coefficient.

2. The method according to claim 1, characterized in that The method further includes: At least one third image is input into the iteratively adjusted depth image prediction model, and a third initial depth image corresponding to the third image is output.

3. The method according to claim 1, characterized in that The method further includes: The adjacent third image and fourth image and the motion data recorded in the time period between shooting the third image and the fourth image are input into the iteratively adjusted motion prediction model, and the motion matrix is ​​output, wherein the motion matrix is ​​used to determine the motion trajectory of the device shooting the third image and the fourth image.

4. The method according to claim 1, characterized in that The motion prediction model includes a first motion encoder PoseNet, a second motion encoder BiLSTM, and a gated weighted network; the depth image prediction model includes an image encoder DepthNnet.

5. The method according to claim 4, characterized in that The determining of the first motion feature according to the adjacent first image and the second image specifically includes: stacking the adjacent first image and second image; Extracting first motion features of the adjacent first and second images of the stack according to the first motion encoder PoseNet.

6. The method according to claim 4, characterized in that Determining the second motion feature according to the IMU data specifically includes: The second motion feature corresponding to the IMU data is determined by the second motion encoder BiLSTM.

7. The method according to claim 1, characterized in that Determining a first weight coefficient corresponding to the first motion feature according to a gated weighted network specifically includes: The first motion feature is passed through a fully connected network to determine a first intermediate parameter; A first weight coefficient corresponding to the first motion feature is determined according to the first intermediate parameter and a Sigmoid function.

8. The method according to claim 1, characterized in that Determining a second weight coefficient corresponding to the second motion feature according to a gated weighted network specifically includes: The second motion feature is passed through a fully connected network to determine a second intermediate parameter; A second weight coefficient corresponding to the second motion feature is determined according to the second intermediate parameter and the Sigmoid function.

9. The method according to claim 4, characterized in that The IMU data includes three-dimensional acceleration data and three-dimensional gyroscope data.

10. The method according to claim 1, characterized in that The method further includes: Determine, according to the two-dimensional coordinates of any pixel point in the first image, the three-dimensional coordinates of the any pixel point in the first initial depth image corresponding to the first image; According to the three-dimensional coordinates and the motion matrix, a first parameter of reprojecting the first image to the adjacent second image and a second parameter of reprojecting the first depth image corresponding to the first image to the adjacent second depth image corresponding to the second image are determined.

11. The method according to claim 10, characterized in that The determining the loss function according to the first initial depth image, the second initial depth image and the motion matrix specifically includes: The loss function is determined according to the first parameter and the second parameter.

12. The method according to claim 1, characterized in that The loss functions include an image reconstruction loss function, a depth consistency loss function, and a depth smoothing loss function.

13. A device for adjusting a depth image prediction model, characterized in that: The device includes: An acquisition unit, used for acquiring a sequence of sequentially captured images; The acquisition unit is further used to acquire a plurality of sample data groups, each sample data group comprising a first image and a second image adjacent to each other in an image sequence, and motion data recorded in a time period between taking the first image and the second image, wherein the motion data comprises inertial measurement unit IMU data; An adjustment unit, configured to iteratively adjust the parameters of the depth image prediction model and the motion prediction model in an unsupervised manner so as to optimize the loss function; wherein the step of determining the loss function is processed by a determination unit, and the determination unit is specifically configured to: Based on the motion prediction model, determining a motion matrix according to the sample data group; Based on the depth image prediction model, predicting a first initial depth image corresponding to the first image and a second initial depth image corresponding to the second image; Determine the loss function according to the first initial depth image, the second initial depth image and the motion matrix; The determining unit is further configured to: determining a first motion feature according to the adjacent first image and the second image; determining a second motion feature based on the IMU data; The first motion feature and the second motion feature are selectively fused to determine the motion matrix, wherein a first weight coefficient corresponding to the first motion feature is determined according to a gated weighted network, and a second weight coefficient corresponding to the second motion feature is determined according to the gated weighted network; the motion matrix is ​​determined according to the first motion feature, the first weight coefficient, the second motion feature, and the second weight coefficient.

14. A computer-readable storage medium storing computer program instructions, characterized in that: The computer program instructions implement the method according to any one of claims 1 to 12 when executed by a processor.

15. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the steps according to any one of claims 1-12.

Images