Training method of motion state estimation model, motion state estimation method and device

Abstract

The disclosed embodiment discloses a training method for a motion state estimation model, a motion state estimation method and a device. The method includes: using a feature extraction network, based on a first sample image taken at a first moment and a second sample image taken at a second moment, obtaining a first image feature corresponding to the first sample image and a second image feature corresponding to the second sample image; using the first image feature and the second image feature, performing feature fusion to obtain a first fused feature; using a motion state prediction network to predict the first fused feature to obtain a predicted motion state flow of the target object in the sample image set; determining a model loss based on the predicted motion state flow and a reference motion state flow; and training a motion state estimation model based on the model loss. In this way, the trained motion state estimation model can directly use multiple images taken by a vehicle-mounted camera to estimate the motion state of the target object, thereby improving the efficiency of estimating the motion state of the target object.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a motion state estimation model training method, motion state estimation method, device, storage medium and electronic device. Background Art

[0002] In an autonomous driving scenario, the autonomous driving system can use multiple images of the vehicle's surroundings taken at adjacent moments to estimate the motion state of target objects around the vehicle.

[0003] At present, the autonomous driving system needs to first use the target tracking algorithm to correlate and match the target objects in multiple images before it can further calculate the motion state of the target objects. It is impossible to directly obtain the motion state of the target objects from multiple images. Summary of the invention

[0004] Currently, autonomous driving systems are unable to directly obtain the motion state of target objects from multiple images, resulting in low efficiency in estimating the motion state of target objects.

[0005] In order to solve the above technical problems, the present disclosure is proposed. The embodiments of the present disclosure provide a training method for a motion state estimation model, a motion state estimation method and a device.

[0006] According to a first aspect of the present disclosure, a training method for a motion state estimation model is provided, the motion state estimation model includes a feature extraction network and a motion state prediction network, and the training method includes: obtaining a sample image set, the sample image set including a first sample image taken at a first moment and a second sample image taken at a second moment, the first moment being earlier than the second moment; using the feature extraction network, based on the first sample image and the second sample image, obtaining a first image feature corresponding to the first sample image and a second image feature corresponding to the second sample image; using the first image feature and the second image feature, performing feature fusion, feature transformation and feature fusion, and obtaining a first fused feature in a target visual angle space; using the motion state prediction network to predict the first fused feature, and obtaining a predicted motion state flow of the target object in the sample image set; determining the model loss based on the predicted motion state flow and the reference motion state flow; and training the motion state estimation model based on the model loss.

[0007] According to the second aspect of the present disclosure, a motion state estimation method is provided, including: obtaining a set of images to be processed, the set of images to be processed including a first image to be processed taken at a third moment and a second image to be processed taken at a current moment, the third moment being earlier than the current moment; obtaining a motion state estimation model trained by the training method of the motion state estimation model provided by the first aspect of the present disclosure, and obtaining the motion state flow of the target object at the current moment based on the set of images to be processed.

[0008] According to a third aspect of the present disclosure, a training device for a motion state estimation model is provided, the motion state estimation model includes a feature extraction network and a motion state prediction network, and the training device includes: a sample acquisition module, used to acquire a sample image set, the sample image set includes a first sample image taken at a first moment and a second sample image taken at a second moment, the first moment is earlier than the second moment; a feature extraction module, used to use the feature extraction network to obtain a first image feature corresponding to the first sample image and a second image feature corresponding to the second sample image based on the first sample image and the second sample image; a feature fusion module, used to use the first image feature and the second image feature to perform feature fusion, feature transformation and feature fusion, and obtain a first fused feature in a target visual angle space; a prediction module, used to predict the first fused feature using the motion state prediction network, and obtain a predicted motion state flow of the target object in the sample image set; a loss determination module, used to determine the model loss based on the predicted motion state flow and the reference motion state flow; and a training module, used to train the motion state estimation model based on the model loss.

[0009] According to a fourth aspect of the present disclosure, a motion state estimation device is provided, comprising: an image acquisition module, used to acquire a set of images to be processed, the set of images to be processed comprising a first image to be processed taken at a third moment and a second image to be processed taken at a current moment, the third moment being earlier than the current moment; a motion state estimation module, used to obtain a motion state estimation model trained using the training method of the motion state estimation model described in the first aspect above, and to obtain a motion state flow of the target object at the current moment based on the set of images to be processed.

[0010] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor executable instructions; a processor for reading executable instructions from the memory and executing the instructions to implement the above-mentioned motion state estimation model training method or motion state estimation method.

[0011] The technical solution of the disclosed embodiment first uses a feature extraction network to extract a first image feature and a second image feature from a first sample image and a second sample image taken successively; then the first image feature and the second image feature are feature-fused to obtain a first fused feature; next, the first fused feature is predicted using a motion state prediction network to obtain a predicted motion state flow of the target object; based on the predicted motion state flow and the reference motion state flow, the model loss is determined; based on the model loss, the motion state estimation model is trained to obtain a trained motion state estimation model. In this way, the trained motion state estimation model can directly use multiple images taken by the vehicle-mounted camera to estimate the motion state of the target object, thereby improving the efficiency of estimating the motion state of the target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0013] Figure 1 It is a schematic diagram of an exemplary system architecture applicable to various methods and devices of the embodiments of the present disclosure.

[0014] Figure 2 It is a flowchart of a method for training a motion state estimation model provided by an exemplary embodiment of the present disclosure.

[0015] Figure 3 It is a schematic diagram of the algorithm structure of a motion state estimation model provided by an exemplary embodiment of the present disclosure.

[0016] Figure 4 It is a flowchart of step S230 of a method for training a motion state estimation model provided by an exemplary embodiment of the present disclosure.

[0017] Figure 5 It is a flowchart of determining a reference motion state flow provided by an exemplary embodiment of the present disclosure.

[0018] Figure 6 4 is a flowchart of step S330 of determining a reference motion state flow provided by an exemplary embodiment of the present disclosure.

[0019] Figure 7 4 is a flowchart of step S320 of determining a reference motion state flow provided by an exemplary embodiment of the present disclosure.

[0020] Figure 8 It is a data processing block diagram for generating a reference motion state stream from point cloud data provided by an exemplary embodiment of the present disclosure.

[0021] Fig. 9 is a flowchart of a motion state estimation method provided by an exemplary embodiment of the present disclosure.

[0022] Fig.10 It is a structural diagram of a training device for a motion state estimation model provided by an exemplary embodiment of the present disclosure.

[0023] Fig.11 is another structural diagram of a training device for a motion state estimation model provided by an exemplary embodiment of the present disclosure.

[0024] Fig.12is a structural diagram of a motion state estimation device provided by an exemplary embodiment of the present disclosure.

[0025] Fig.13 is a structural block diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0027] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0028] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0029] It should also be understood that in the embodiments of the present disclosure, “plurality” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0030] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0031] In addition, the term "and / or" in the present disclosure is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present disclosure generally indicates that the associated objects before and after are in an "or" relationship.

[0032] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0033] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0034] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0035] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0036] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0037] The disclosed embodiments can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0038] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0039] Application Overview

[0040] In an autonomous driving scenario, the autonomous driving system can use the on-board camera to capture the surrounding environment and estimate the motion state of each target object in the surrounding environment based on the captured image. The target object may be, for example, a pedestrian, other vehicles, or an animal.

[0041] At present, the method for estimating the motion state of a target object generally includes the following steps: first, the boundaries of each target object, such as a 3D box, are estimated from the images taken by the vehicle-mounted camera at two adjacent moments; then, the target tracking algorithm is used to match each 3D box in the two images to determine the matching relationship between each target object in the two images; finally, based on the motion state of the vehicle itself and the matching relationship between the target object, the motion state of the target object, such as: motion direction, speed, angular velocity, etc., is calculated.

[0042] It can be seen that the current autonomous driving system cannot directly obtain the motion state of the target object from the multiple images taken by the on-board camera, resulting in low efficiency in estimating the motion state of the target object.

[0043] Exemplary Systems

[0044] Figure 1 It is a schematic diagram of an exemplary system architecture applicable to various methods and devices of the embodiments of the present disclosure.

[0045] like Figure 1 As shown, the system architecture 100 may include a terminal device 101 , a network 102 , a server 103 and a collection device 104 .

[0046] The terminal device 101 may be various electronic devices, including but not limited to mobile phones, laptop computers, tablet computers PAD, vehicle terminals (such as vehicle navigation terminals), autonomous driving computing platforms, vehicle computing modules, mobile data centers (MDC), etc. Various communication client applications may be installed on the terminal device 101, such as navigation applications, electronic map applications, search applications, web browser applications, instant messaging tools, etc. In some implementations, the terminal device 101 may be set on various mobile objects, such as vehicles (cars), ships, trains, subways, airplanes, etc., as well as various robots, such as service robots, transport robots, automated guided vehicles (AGVs), unmanned ground vehicles (UGVs), etc.

[0047] The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include networks of various modes such as 2G, 3G, 4G, and 5G, and is used to enable the terminal device 101 to communicate with the server 103 through the network 102 to receive or send messages, etc., for example, to realize vehicle-to-everything (V2X) of the vehicle.

[0048] The server 103 may be a server that provides various services, for example, it may be an artificial intelligence (AI) server that uses the sample image set collected by the acquisition device 104 to perform model training, or uses the image set to be processed collected by the acquisition device 104 to predict the motion state of the target object. Exemplarily, the AI ​​server may use the sample image set to perform model training to obtain a motion state estimation model, or use the motion state estimation model to predict the received image set to be processed to obtain the motion state of the target object. It should be noted that the server 103 in the present disclosure may be a backend server provided by the manufacturer of the terminal device or the service provider of the autonomous driving system, or it may be a cloud server provided by an Internet service provider (ISP) or a cloud service provider, or it may be a distributed server cluster deployed at different network nodes. The embodiments of the present disclosure do not specifically limit the implementation method of the server 103.

[0049] The acquisition device 104 may be any device for acquiring image or point cloud data, such as a vehicle-mounted camera and / or a vehicle-mounted radar.

[0050] The vehicle-mounted camera can be used to shoot the surroundings of the vehicle to obtain a sample image set or a set of images to be processed. The acquisition device 104 may include one or more vehicle-mounted cameras, and the multiple vehicle-mounted cameras may be set at different positions of the vehicle and face different directions to shoot different directions around the vehicle. When the acquisition device 104 includes multiple vehicle-mounted cameras, the acquisition device 104 can obtain multiple images taken by different vehicle-mounted cameras at the same time. The vehicle-mounted camera can be a fisheye camera, a narrow-angle camera, etc., and the embodiment of the present disclosure does not limit the type of the vehicle-mounted camera.

[0051] The vehicle-mounted radar can be used to scan the environment around the vehicle to obtain a point cloud data set corresponding to the environment around the vehicle. The acquisition device 104 may include one or more vehicle-mounted radars, and the multiple vehicle-mounted radars may be set at different positions of the vehicle and facing different directions to scan different directions around the vehicle. The vehicle-mounted radar may be a millimeter wave radar, a laser radar, an ultrasonic radar, an infrared radar, etc., and the embodiment of the present disclosure does not limit the type of the vehicle-mounted radar.

[0052] It should be noted that the training method of the motion state estimation model or the motion state estimation method provided in the embodiments of the present disclosure can be executed by the terminal device 101 or by the server 103. Accordingly, the training device of the motion state estimation model or the motion state estimation device can be set in the terminal device 101 or in the server 103.

[0053] It should be understood that Figure 1The number of terminal devices, networks and servers in the system is only illustrative. According to the implementation requirements, there may be any number of terminal devices, networks and servers. In the case where the sample image set for model training and the image set to be processed for motion state estimation do not need to be acquired remotely, the above system architecture may not include a network, but only include the terminal device 101 and the acquisition device 104.

[0054] Exemplary Methods

[0055] Figure 2 It is a flowchart of a method for training a motion state estimation model provided by an exemplary embodiment of the present disclosure.

[0056] This embodiment can be applied to electronic devices (for example Figure 1 The terminal device 101 or server 103 shown in the figure). A motion state estimation model to be trained may be deployed in the electronic device, and the motion state estimation model to be trained may include one or more neural networks, for example, one or more feature extraction networks, and one or more motion state prediction networks.

[0057] like Figure 2 As shown, the training method of the motion state estimation model may include the following steps:

[0058] Step S210 , obtaining a sample image set, the sample image set comprising a first sample image taken at a first moment and a second sample image taken at a second moment, the first moment being earlier than the second moment.

[0059] The sample image set may be obtained by taking pictures of the surroundings of the vehicle by the vehicle-mounted camera. The sample image set may include multiple sample images taken at different times. In the embodiment of the present disclosure, the first moment and the second moment refer to any two different moments that come one after another. The first sample image and the second sample image refer to images taken at any two different moments that come one after another. Among them, when the number of vehicle-mounted cameras is multiple, the first sample image may include multiple images taken by multiple vehicle-mounted cameras at the first moment, and the second sample image may include multiple images taken by multiple vehicle-mounted cameras at the second moment.

[0060] For example, the vehicle-mounted camera can capture sample images at a specific time interval Δt, for example, capture a sample image at time t-1, and then capture another sample image at time t after an interval of Δt. Then, for the images captured at time t-1 and time t, the sample image captured at time t-1 is the first sample image I t-1 , the sample image captured at time t is the second sample image I t .

[0061] For example, the vehicle-mounted camera can capture a video stream at a certain frame rate. Then, the first sample image and the second sample image can be any two adjacent sample images in the video stream. For example: the first sample image I t-1 It can be the previous sample image of two adjacent sample images taken by the vehicle-mounted camera at time t-1, and the second sample image I t It may be the latter frame of sample image of two adjacent frames of sample images taken by the vehicle-mounted camera at time t.

[0062] Exemplarily, the electronic device may also perform frame sampling on the video stream captured by the vehicle-mounted camera, and select one frame of image from the video stream every n frames as a sample image. For example, the mth frame of sample image captured by the vehicle-mounted camera at time t-1 is used as the first sample image, and the m+nth frame of sample image captured by the vehicle-mounted camera at time t is used as the second sample image.

[0063] Step S220: using a feature extraction network, based on the first sample image and the second sample image, obtain a first image feature corresponding to the first sample image and a second image feature corresponding to the second sample image.

[0064] The first image feature is the image feature of the first sample image in the camera viewing angle space, and the second image feature is the image feature of the second sample image in the camera viewing angle space. The first image feature and the second image feature include, but are not limited to, texture features, edge contour features, category features, grayscale features, histogram features, etc. of each target object in the sample image, which are not specifically limited here.

[0065] Among them, the feature extraction network can be composed of one or more neural networks, such as: convolutional neural network CNN, deep residual network resnet, recurrent neural network RNN, Transforme network, densenet, mobilenet, BERT network, etc. Each neural network can include one or more neural network layers, such as: convolutional layer, pooling layer, fully connected layer, etc., which are not specifically limited here.

[0066] In a specific implementation, the electronic device can input the first sample image and the second sample image into a feature extraction network, and the feature extraction network can use its internal neural network layer to perform convolution, pooling, and full connection operations on the first sample image and the second sample image to obtain the first image feature and the second image feature.

[0067] Step S230, using the first image feature and the second image feature, perform feature transformation and feature fusion to obtain a first fusion feature in the target viewing angle space.

[0068] The target viewing angle space may be a bird's eye views (BEV) space, or other viewing angle spaces, which is not limited in the embodiments of the present disclosure.

[0069] It can be understood that since the first sample image and the second sample image are images taken by the camera, the first image feature and the second image feature are image features in the camera's viewing angle space, and the coordinate system they use is the image coordinate system in the camera's viewing angle space. When there are multiple on-board cameras, the image coordinate systems corresponding to different cameras are also different, which is not conducive to feature fusion. In addition, autonomous driving scenarios also usually require estimating the motion state of the target object in the BEV space. Therefore, in order to facilitate feature fusion and subsequent estimation of the motion state of the target object in the BEV space, the electronic device can use feature transformation to transfer the first image feature and the second image feature to the same target viewing angle space, such as the BEV space, and then fuse the converted first image feature and the second image feature to obtain a first fused feature.

[0070] In the embodiment of the present disclosure, feature fusion can be implemented by concat fusion, add fusion, or other fusion methods, which is not limited in the embodiment of the present disclosure.

[0071] Step S240: predict the first fusion feature using the motion state prediction network to obtain a predicted motion state flow of the target object in the sample image set.

[0072] Among them, the motion state prediction network can be composed of one or more neural networks, such as: a recurrent neural network, a long short-term memory network LSTM, a gated recurrent unit network GRU, etc., which is not specifically limited here.

[0073] The electronic device can use a machine learning method to input the first fusion feature into a motion state prediction network to be trained, predict the target object in the sample image set through the motion state prediction network, and obtain a predicted motion state flow of the target object.

[0074] In one implementation, the predicted motion state flow may be a tensor in the target visual space used to characterize the predicted motion state value of the target object. The size of the tensor may be, for example, H*W*3. Wherein: H and W represent the size of the target visual space; 3 represents three parameters of the predicted value of the motion state of the target object, which are the predicted value of the displacement of the target object along the x direction from the first moment to the second moment, the predicted value of the displacement of the target object along the y direction from the first moment to the second moment, and the predicted value of the change in the orientation angle of the target object from the first moment to the second moment. Wherein, the x direction is the front-to-back direction of the vehicle, and the y direction is the left-to-right direction of the vehicle.

[0075] For example, assuming that the position (i, j) in the target viewing space is within a target object, then the parameter value of the position (i, j, 0) in the predicted motion state flow represents the predicted value of the displacement of the target object along the x-direction from the first moment to the second moment, the parameter value of the position (i, j, 1) represents the predicted value of the displacement of the target object along the y-direction from the first moment to the second moment, and the parameter value of the position (i, j, 2) represents the predicted value of the change in the orientation angle of the target object from the first moment to the second moment.

[0076] It can be understood that if the position (i, j) in the predicted motion state flow does not belong to any target object, then the values ​​of (i, j, 0), (i, j, 1), and (i, j, 2) are all 0.

[0077] Step S250, determining the model loss based on the predicted motion state flow and the reference motion state flow.

[0078] In one implementation, the reference motion state flow may be a tensor in the target visual angle space used to characterize the true value of the motion state of the target object. The reference motion state flow is preferably of the same size as the tensor of the predicted motion state flow, for example, H*W*3, for ease of calculation. Wherein: H and W represent the size of the target visual angle space; 3 represents three parameters of the true value of the motion state of the target object, which are respectively the true value of the displacement of the target object along the x direction from the first moment to the second moment, the true value of the displacement of the target object along the y direction from the first moment to the second moment, and the true value of the change in the orientation angle of the target object from the first moment to the second moment.

[0079] For example, assuming that the position (i, j) in the target viewing space is within a target object, then the parameter value of the position (i, j, 0) in the reference motion state flow represents the true value of the displacement of the target object along the x-direction from the first moment to the second moment, the parameter value of the position (i, j, 1) represents the true value of the displacement of the target object along the y-direction from the first moment to the second moment, and the parameter value of the position (i, j, 2) represents the true value of the change in the orientation angle of the target object from the first moment to the second moment.

[0080] It can be understood that if the position (i, j) in the reference motion state flow does not belong to any target object, then the values ​​of (i, j, 0), (i, j, 1), and (i, j, 2) are all 0.

[0081] In the disclosed embodiment, the electronic device may determine the model loss based on any loss function such as L1 loss function, L2 loss function, Smooth L1 loss function, cross entropy loss function, log loss function, maximum likelihood loss function, etc. Taking the L1 loss function as an example, the model loss Loss may be:

[0082] Loss=|motion_flow pred -motion_flow gt |

[0083] Among them, motion_flow pred Indicates the predicted motion state flow, motion_flow gt Represents the base motion state flow. |motion_flow pred -motion_flow gt | represents the absolute value of the error between the predicted motion state flow and the reference motion state flow.

[0084] Step S260: training the motion state estimation model based on the model loss.

[0085] Based on the network loss, the motion state estimation model can use the reference motion state flow as the target of its prediction output and perform multiple iterations, thereby continuously updating its internal model parameters and making the model loss continuously converge. Among them, when the model loss converges to a preset condition, for example, when the value of the model loss Loss is less than a preset threshold, the training is terminated to obtain a trained motion state estimation model, or when the training iteration of the motion state prediction network reaches a preset number of times, the training is terminated to obtain a trained motion state estimation model.

[0086] The technical solution of the disclosed embodiment first uses a feature extraction network to extract a first image feature and a second image feature from a first sample image and a second sample image taken successively; then the first image feature and the second image feature are feature-fused to obtain a first fused feature; next, the first fused feature is predicted using a motion state prediction network to obtain a predicted motion state flow of the target object; based on the predicted motion state flow and the reference motion state flow, the model loss is determined; based on the model loss, the motion state estimation model is trained to obtain a trained motion state estimation model. In this way, the trained motion state estimation model can directly use multiple images taken by the vehicle-mounted camera to estimate the motion state of the target object, thereby improving the efficiency of estimating the motion state of the target object.

[0087] Figure 3 It is a schematic diagram of the algorithm structure of a motion state estimation model provided by an exemplary embodiment of the present disclosure.

[0088] like Figure 3 As shown, in one implementation, the feature extraction network of the motion state estimation model may include a first feature extraction network and a second feature extraction network, wherein the first feature extraction network and the second feature extraction network may be the same type of neural network or different types of neural networks.

[0089] based on Figure 3 The motion state estimation model shown in the above Figure 2 Based on the embodiment shown, step S220 may include the following steps:

[0090] Step S221, using a first feature extraction network to obtain a first image feature based on a first sample image.

[0091] In a specific implementation, the electronic device may input the first sample image into the first feature extraction network. The size of the first sample image may be (v*3*h*w), where v represents the number of first sample images, 3 represents the number of channels of the sample image is 3 (for example, RGB channels), h represents the height of the first sample image, and w represents the width of the first sample image. The first feature extraction network may use its internal neural network layer to perform convolution, pooling, and full connection operations on the first sample image to obtain the first image feature.

[0092] The first image feature can be a feature tensor of a preset size output by the first feature extraction network, and the size of the feature tensor can be (v*c*h1*w1), where v represents the number of sample images, c represents the number of channels of the feature tensor, h1 represents the height of the feature tensor, and w1 represents the width of the feature tensor.

[0093] Step S222: using a second feature extraction network to obtain a second image feature based on the second sample image.

[0094] In a specific implementation, the electronic device may input the second sample image into the second feature extraction network. The size of the second sample image may be (v*3*h*w), where v represents the number of second sample images, 3 represents the number of channels of the sample image is 3 (e.g., RGB channels), h represents the height of the second sample image, and w represents the width of the second sample image. The second feature extraction network may use its internal neural network layer to perform convolution, pooling, and full connection operations on the second sample image to obtain the second image feature.

[0095] The second image feature can be a feature tensor of a preset size output by the second feature extraction network, and the size of the feature tensor can be (v*c*h1*w1), where v represents the number of sample images, c represents the number of channels of the feature tensor, h1 represents the height of the feature tensor, and w1 represents the width of the feature tensor.

[0096] It should be noted that in the embodiments of the present disclosure, the sizes of the first sample image and the second sample image can be the same or different, and the sizes of the first image features and the second image features can be the same or different, and the embodiments of the present disclosure do not specifically limit this. Among them, the sizes of the first sample image and the second sample image are the same, and the sizes of the first image features and the second image features are the same, which is a preferred implementation method, so as to facilitate the subsequent feature fusion of the sizes of the first image features and the second image features.

[0097] The technical solution of the embodiment of the present disclosure utilizes two feature extraction networks to extract features from the first sample image and the second sample image respectively, to obtain first image features and second image features, so as to facilitate subsequent feature fusion.

[0098] Figure 4 It is a flowchart of step S230 of a method for training a motion state estimation model provided by an exemplary embodiment of the present disclosure.

[0099] like Figure 4 As shown in the above Figure 2 and Figure 3 Based on the illustrated embodiment, step S230 may include the following steps:

[0100] Step S231, using the first transfer matrix, projecting the first image feature and the second image feature into the target viewing angle space to obtain a first projection feature corresponding to the first image feature in the target viewing angle space and a second projection feature corresponding to the second image feature in the target viewing angle space.

[0101] The first transfer matrix is ​​a homography matrix from the target view coordinate system to the image coordinate system.

[0102] In a specific implementation, the electronic device first uses a first transfer matrix to perform feature projection on the first image feature and the second image feature to obtain a first projection feature corresponding to the first image feature in the BEV space and a second projection feature corresponding to the second image feature in the BEV space.

[0103] The first projection feature and the second projection feature may both be feature tensors of a preset size. The size of the first projection feature and the size of the second projection feature may be the same or different, and the disclosed embodiment does not limit this. To facilitate feature fusion, the sizes of the first projection feature and the second projection feature are preferably the same.

[0104] Exemplarily, the sizes of the first projection feature and the second projection feature can both be (1*c*h2*w2), where c represents the number of channels of the feature tensor, h2 represents the height of the feature tensor, and w2 represents the width of the feature tensor.

[0105] Step S232: transform the first projection feature using the second transfer matrix to obtain a third projection feature corresponding to the first projection feature at the second moment.

[0106] Among them, the second transfer matrix is ​​the vehicle posture change matrix from the first moment to the second moment.

[0107] It is understandable that when the vehicle moves, as the vehicle's posture changes, the BEV space in which the vehicle is located also changes at all times. Therefore, although the first projection feature and the second projection feature are both projection features in the BEV space, they are not projection features in the BEV space at the same time, which is not conducive to direct feature fusion. In this regard, the electronic device uses the second transfer matrix to transform the first projection feature, transforms the first projection feature into the BEV space at the second moment, and obtains the third projection feature. In this way, the first projection feature and the third projection feature are located in the BEV space at the same moment, which is conducive to subsequent feature fusion.

[0108] Exemplarily, the sizes of the third projection feature and the second projection feature are preferably the same, and both can be (1*c*h2*w2), where c represents the number of channels of the feature tensor, h2 represents the height of the feature tensor, and w2 represents the width of the feature tensor.

[0109] Step S233: Fusing the third projection feature with the second projection feature to obtain a first fused feature.

[0110] In a specific implementation, the feature fusion of the second projection feature and the third projection feature can be implemented by concat fusion, add fusion, or other fusion methods, which is not limited in the embodiments of the present disclosure. The size of the first fused feature can be the same as or different from the size of the second projection feature or the third projection feature, which may be related to the fusion method. For example: when concat fusion is used, the number of channels of the first fused feature can be doubled compared to the number of channels of the second projection feature or the third projection feature, and the size can reach (1*2c*h2*w2).

[0111] The solution of the embodiment of the present disclosure converts the first image feature and the second image feature in the camera perspective space at different moments into the BEV space at the same moment, and performs feature fusion, so that the first fused feature can express the changes of the image features of the target object over time in the BEV space.

[0112] Figure 5 It is a flowchart of determining a reference motion state flow provided by an exemplary embodiment of the present disclosure.

[0113] like Figure 5 As shown in the above Figure 2Based on the illustrated embodiment, determining the reference motion state flow may include the following steps:

[0114] Step S310 , obtaining a point cloud data set, where the point cloud data set includes first point cloud data collected at a first moment and second point cloud data collected at a second moment.

[0115] The point cloud data set may be obtained by scanning the vehicle surroundings by the vehicle-mounted radar. The point cloud data set may include multiple point cloud data scanned by the vehicle-mounted radar at different times. In the disclosed embodiment, the first point cloud data and the second point cloud data refer to the point cloud data scanned at any two different times in the point cloud data set.

[0116] For example, the vehicle-mounted radar can scan the vehicle's surrounding environment at a specific time interval Δt, for example: perform a scan at time t-1 to obtain a point cloud data, and then perform another scan at time t after an interval of Δt to obtain another point cloud data. Then, for the point cloud data scanned at time t-1 and time t, the point cloud data scanned at time t-1 is the first point cloud data P t-1 , the point cloud data scanned at time t is the second point cloud data P t .

[0117] Step S320: using a target detection algorithm, determine first position information of the target object in the first point cloud data and second position information of the target object in the second point cloud data.

[0118] The first orientation information may include the position information and orientation information of the target object in the radar coordinate system at the first moment, and the second orientation information may include the position information and orientation information of the target object in the radar coordinate system at the second moment. For example, the position information may include the three-dimensional coordinates (x, y, z) of the target object in the radar coordinate system, wherein the x direction of the radar coordinate system is the front-to-back direction of the vehicle, the y direction is the left-to-right direction of the vehicle, and the z direction is the height direction of the vehicle. The position information may also include the three-dimensional size (w, h, l) of the target object, wherein w, h, and l represent the width, height, and length of the target object, respectively. The orientation information may include the angle yaw between the orientation of the target object and the x direction.

[0119] In one implementation, the orientation information of the target object can be described by a bounding box (BBox). Then, for any target object k, its bounding box can be expressed as:

[0120]

[0121] For the sake of distinction, the first orientation information described by the bounding box is hereinafter referred to as BBox in the present disclosure. t-1, the second orientation information described by the bounding box is recorded as BBox t .

[0122] Step S330: Determine the change information of the orientation of the target object from the first moment to the second moment by using the first orientation information and the second orientation information.

[0123] In a specific implementation, the electronic device may transfer the first orientation information and the second orientation information to the same target visual field space, such as the BEV space at the same time, so as to compare the position change and orientation change of the target object at the first time and the second time. Then, in the BEV space, the orientation change information of the target object from the first time to the second time is determined, wherein the orientation change information may include the position change amount and orientation change amount of the target object from the first time to the second time. The position change amount may include the displacement dx of the target object along the x direction and the displacement dy along the y direction, and the orientation change amount may include the angle change d between the orientation of the target object and the x direction. yaw .

[0124] Step S340: Update the initial motion state flow using the orientation change information to obtain a reference motion state flow.

[0125] In a specific implementation, the electronic device may initialize a motion state stream. The size of the initial motion state stream is preferably the same as the size of the predicted motion state stream, for example, both are H*W*3. The values ​​of the parameters in the initial motion state stream are all 0. Next, for any target object, the electronic device may store its position change information (for example: dx, dy, d yaw ) is updated to the position corresponding to the target object in the initial motion state flow to obtain the reference motion state flow of the target object.

[0126] The technical solution of the disclosed embodiment first uses a target detection algorithm to extract the first orientation information and the second orientation information of the target object from the first point cloud data and the second point cloud data scanned successively; then, the first orientation information and the second orientation information are used to determine the orientation change information of the target object from the first moment to the second moment; finally, the orientation change information is used to update the initial motion state flow to obtain the reference motion state flow. This provides a training target for the motion state estimation model, which is conducive to improving the training effect of the motion state estimation model.

[0127] Figure 6 4 is a flowchart of step S330 of determining a reference motion state flow provided by an exemplary embodiment of the present disclosure.

[0128] like Figure 6 As shown in the above Figure 5 Based on the embodiment shown, step S330 may include the following steps:

[0129] Step S331, using the third transfer matrix, projecting the first orientation information and the second orientation information into the target viewing angle space to obtain first projection orientation information corresponding to the first orientation information in the target viewing angle space and second projection orientation information corresponding to the second orientation information in the target viewing angle space.

[0130] The third transfer matrix is ​​an external parameter matrix from the vehicle-mounted radar coordinate system to the target viewing angle coordinate system. The target viewing angle coordinate system may be a BEV coordinate system or other coordinate systems, which is not limited in the embodiments of the present disclosure.

[0131] It is understandable that since the first point cloud data and the second point cloud data are point cloud data acquired by the vehicle-mounted radar, the first orientation information and the second orientation information are orientation information in the vehicle-mounted radar coordinate system. However, autonomous driving scenarios usually require estimating the motion state of the target object in the BEV space, so it is necessary to obtain the reference motion state flow in the BEV space. In this regard, the electronic device first uses the third transfer matrix to project the first orientation information and the second orientation information to obtain the first projection orientation information corresponding to the first orientation information in the BEV space and the second projection orientation information corresponding to the second orientation information in the BEV space.

[0132] Step S332: transform the first projection orientation information using the second transfer matrix to obtain third projection orientation information corresponding to the first projection orientation information at the second moment.

[0133] It is understandable that when the vehicle moves, as the vehicle's posture changes, the BEV coordinate system in which the vehicle is located also changes at all times. Therefore, although the first projection orientation information and the second projection orientation information are both orientation information in the BEV space, they are not orientation information in the BEV space at the same time, which is not conducive to directly calculating the orientation change information of the target object. In this regard, the electronic device uses the second transfer matrix to transform the first projection orientation information, transforms the first projection orientation information into the BEV space at the second moment, and obtains the third projection orientation information. In this way, the first projection orientation information and the third projection orientation information are located in the BEV space at the same moment, which is conducive to the subsequent orientation change information of the target object.

[0134] Step S333: Determine the orientation change information using the second projection orientation information and the third projection orientation information.

[0135] In a specific implementation, the electronic device may calculate the orientation of the target object in the second projection orientation information and the third projection orientation information to determine the orientation change information of the target object from the first moment to the second moment.

[0136] The solution provided by the embodiment of the present disclosure converts the first orientation information and the second orientation information in the radar coordinate system at different times into the BEV space at the same time, so as to more accurately determine the orientation change information of the target object.

[0137] Figure 7 4 is a flowchart of step S320 of determining a reference motion state flow provided by an exemplary embodiment of the present disclosure.

[0138] It is understandable that when a vehicle is traveling on a road, there may be multiple target objects around it, such as multiple other vehicles, multiple pedestrians, etc. In this case, the vehicle needs to determine the position change information of each target object separately.

[0139] like Figure 7 As shown, in order to determine the orientation change information of each target object respectively, in the above Figure 5 Based on the embodiment shown, step S330 may include the following steps:

[0140] Step S321: using a target tracking algorithm, determine a correspondence between first position information and second position information of each target object among a plurality of target objects.

[0141] In a specific implementation, for multiple target objects, the electronic device may establish a first position information set corresponding to a first moment and a second position information set corresponding to a second moment.

[0142] The first position information set may include the first position information corresponding to each target object at the first moment, for example:

[0143]

[0144] Among them, A t-1 represents the first position information set, n is the number of target objects, is the first position information of n target objects.

[0145] The second position information set may include the second position information corresponding to each target object at the second moment, for example:

[0146]

[0147] Among them, A t represents the second position information set, n is the number of target objects, is the second position information of n target objects.

[0148] Next, the electronic device can execute a target tracking algorithm on the first position information set and the second position information set to determine the corresponding relationship between each first position information in the first position information set and each second position information in the second position information set. The matching relationship here refers to which first position information in the first position information set and which second position information in the second position information belong to the same target object.

[0149] Among them, the target tracking algorithm includes but is not limited to a feature point-based target tracking algorithm, a grayscale-based target tracking algorithm or other target tracking algorithms, such as: SIFT algorithm, SURF algorithm, PCA-SIFT algorithm, Moravec algorithm, Susan algorithm, etc., which is not limited to the embodiments of the present disclosure.

[0150] Step S322: Based on the corresponding relationship, the first orientation information and the second orientation information of each target object are respectively used to determine the orientation change information of each target object from the first moment to the second moment.

[0151] In a specific implementation, the electronic device may respectively execute steps S331 to S333 for each target object to determine the orientation change information of the target object, which will not be described in detail here.

[0152] Figure 8 It is a data processing block diagram for generating a reference motion state stream from point cloud data provided by an exemplary embodiment of the present disclosure.

[0153] like Figure 8 As shown, based on the above Figure 5-Figure 7 In the embodiment shown, in order to generate the reference motion state stream, the electronic device may first use the target detection algorithm to perform 3D target detection on the first point cloud data and the second point cloud data to obtain the first orientation information of each target object in the plurality of target objects (for example, the bounding box BBox t-1 ) and second orientation information (e.g., bounding box BBox t ). Next, the electronic device can use the target tracking algorithm to track each target object to determine the corresponding relationship between each target object in the first point cloud data and the second point cloud data. In addition, the electronic device can use the third transfer matrix to project the first orientation information and the second orientation information into the BEV space to obtain the first projection orientation information and the second projection orientation information. Next, the electronic device can use the second transfer matrix to convert the first projection orientation information to the second moment to obtain the third projection orientation information. Finally, the electronic device can compare the second projection orientation information and the third projection orientation information of each target object based on the target tracking results of each target object to obtain a reference motion state flow.

[0154] The solution of the embodiment of the present disclosure, for a scenario where there are multiple target objects around a vehicle, uses a target tracking algorithm to match the correspondence between the first position information and the second position information of the multiple target objects, and uses the correspondence to determine the position change information of each target object respectively, so that the motion state estimation model can have the ability to simultaneously distinguish and track the motion states of multiple target objects.

[0155] Fig. 9 1 is a flow chart of a motion state estimation method provided by an exemplary embodiment of the present disclosure. The method can be applied to electronic devices and can predict the motion state of a target object by using the motion state estimation model trained by the schemes of the above embodiments. Fig. 9 As shown, the method includes:

[0156] Step S410, obtaining a set of images to be processed, the set of images to be processed comprising a first image to be processed taken at a third moment and a second image to be processed taken at a current moment, the third moment being earlier than the current moment.

[0157] The image set to be processed may be obtained by real-time photography of the surroundings of the vehicle by the vehicle-mounted camera during the driving process of the vehicle. The image set to be processed may include a first image to be processed taken by the vehicle-mounted camera at a third moment before the current moment, and a second image to be processed taken at the current moment.

[0158] Exemplarily, the vehicle-mounted camera can capture sample images at a specific time interval Δt. Then, the second image to be processed may be an image captured by the vehicle-mounted camera at the current time t0, and the first image to be processed may be an image captured by the vehicle-mounted camera at time t0-Δt before the current time t0.

[0159] For example, the vehicle-mounted camera can shoot a video stream at a certain frame rate. Then, the second image to be processed can be a current frame image shot by the vehicle-mounted camera at the current moment, and the first image to be processed can be a frame image before the current frame image shot by the vehicle-mounted camera.

[0160] Step S420: using the motion state estimation model, based on the image set to be processed, obtain the motion state flow of the target object at the current moment.

[0161] In a specific implementation, the electronic device can input the image to be processed into a trained motion state estimation model, use the motion state estimation model to predict the image set to be processed, and output the motion state stream of the target object at the current moment.

[0162] According to the technical solution provided by the embodiments of the present disclosure, the electronic device can use the motion state estimation model to directly estimate the motion state of the target object from multiple images taken by the vehicle-mounted camera, thereby improving the efficiency of estimating the motion state of the target object.

[0163] Exemplary Devices

[0164] Fig.10 1 is a structural diagram of a training device for a motion state estimation model provided by an exemplary embodiment of the present disclosure. Exemplarily, the training device for the motion state estimation model can be set in an electronic device to execute the training method for the motion state estimation model of any of the above embodiments of the present disclosure.

[0165] like Fig.10 As shown, the image processing device includes: a sample acquisition module 510, a feature extraction module 520, a feature fusion module 530, a prediction module 540, a loss determination module 550, and a training module 560.

[0166] in:

[0167] The sample acquisition module 510 is used to acquire a sample image set, where the sample image set includes a first sample image taken at a first moment and a second sample image taken at a second moment, where the first moment is earlier than the second moment.

[0168] The feature extraction module 520 is used to obtain a first image feature corresponding to the first sample image and a second image feature corresponding to the second sample image based on the first sample image and the second sample image by using a feature extraction network.

[0169] The feature fusion module 530 is used to perform feature fusion using the first image feature and the second image feature to obtain a first fused feature.

[0170] The prediction module 540 is used to predict the first fusion feature using the motion state prediction network to obtain the predicted motion state flow of the target object in the sample image set.

[0171] The loss determination module 550 is used to determine the model loss based on the predicted motion state flow and the reference motion state flow.

[0172] The training module 560 is used to train the motion state estimation model based on the model loss.

[0173] Fig.11 is another structural diagram of a training device for a motion state estimation model provided by an exemplary embodiment of the present disclosure.

[0174] In one embodiment, Fig.11 As shown, the feature extraction module 520 specifically includes:

[0175] The first feature extraction unit 521 is used to obtain a first image feature based on a first sample image by using a first feature extraction network.

[0176] The second feature extraction unit 522 is used to obtain a second image feature based on the second sample image by using a second feature extraction network.

[0177] In one embodiment, Fig.11 As shown, the feature fusion module 530 specifically includes:

[0178] The first projection unit 531 is used to project the first image feature and the second image feature into the target viewing angle space using the first transfer matrix to obtain the first projection feature corresponding to the first image feature in the target viewing angle space and the second projection feature corresponding to the second image feature in the target viewing angle space. The first transfer matrix is ​​a homography matrix from the target viewing angle coordinate system to the image coordinate system.

[0179] The second projection unit 532 is used to transform the first projection feature by using a second transfer matrix to obtain a third projection feature corresponding to the first projection feature at the second moment, and the second transfer matrix is ​​a vehicle posture change matrix from the first moment to the second moment.

[0180] The fusion unit 533 is used to perform feature fusion on the third projection feature and the second projection feature, perform feature transformation and feature fusion, and obtain a first fusion feature in the target viewing angle space.

[0181] In one embodiment, Fig.11 As shown, the training set of the motion state estimation model also includes: a point cloud acquisition module 610, a target detection module 620, a determination module 630, and a generation module 640. Among them:

[0182] The point cloud acquisition module 610 is used to acquire a point cloud data set, where the point cloud data set includes first point cloud data collected at a first moment and second point cloud data collected at a second moment.

[0183] The target detection module 620 is used to determine the first position information of the target object in the first point cloud data and the second position information of the target object in the second point cloud data by using a target detection algorithm.

[0184] The determination module 630 is used to determine the change information of the orientation of the target object from the first moment to the second moment by using the first orientation information and the second orientation information.

[0185] The generation module 640 is used to update the initial motion state flow using the position change information to obtain a reference motion state flow.

[0186] In one embodiment, Fig.11 As shown, the determination module 630 specifically includes:

[0187] The third projection unit 631 is used to project the first orientation information and the second orientation information into the target viewing angle space using the third transfer matrix to obtain the first projection orientation information corresponding to the first orientation information in the target viewing angle space and the second projection orientation information corresponding to the second orientation information in the target viewing angle space. The third transfer matrix is ​​an external parameter matrix from the vehicle-mounted radar coordinate system to the target viewing angle coordinate system.

[0188] The fourth projection unit 632 is used to transform the first projection orientation information by using the second transfer matrix to obtain third projection orientation information corresponding to the first projection orientation information at the second moment.

[0189] The determination unit 633 is used to determine the orientation change information by using the second projection orientation information and the third projection orientation information.

[0190] In one embodiment, Fig.11 As shown, the determination module 630 also includes:

[0191] The target tracking unit 634 is used to determine the correspondence between the first position information and the second position information of each target object among the multiple target objects by using a target tracking algorithm.

[0192] The determination unit 633 is further configured to determine the orientation change information of each target object from the first moment to the second moment by using the first orientation information and the second orientation information of each target object respectively based on the corresponding relationship.

[0193] Fig.12 1 is a structural diagram of a motion state estimation device provided by an exemplary embodiment of the present disclosure. Exemplarily, the motion state estimation device can be arranged in an electronic device to execute the motion state estimation method of any of the above embodiments of the present disclosure.

[0194] like Fig.12 As shown, the motion state estimation device includes: an image acquisition module 710, and a motion state estimation module 720. Among them:

[0195] The image acquisition module 710 is used to acquire a set of images to be processed, the set of images to be processed includes a first image to be processed taken at a third moment and a second image to be processed taken at a current moment, the third moment being earlier than the current moment.

[0196] The motion state estimation module 720 is used to obtain the motion state flow of the target object at the current moment based on the image set to be processed by using the motion state estimation model trained by the motion state estimation model training method.

[0197] Exemplary Electronic Devices

[0198] Fig.13is a structural block diagram of an electronic device provided by an exemplary embodiment of the present disclosure.

[0199] like Fig.13 As shown, the electronic device 10 includes one or more processors 111 and a memory 112 .

[0200] The processor 111 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0201] The memory 112 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on a computer-readable storage medium, and the processor 111 may run the program instructions to implement the training method of the motion state estimation model of each embodiment of the present disclosure above, the motion state estimation method, and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0202] In one example, the electronic device 10 may further include: an input device 113 and an output device 114 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0203] In addition, the input device 113 may also include, for example, a keyboard, a mouse, and the like.

[0204] The output device 114 can output various information to the outside, including determined distance information, direction information, etc. The output device 114 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0205] Of course, to simplify, Fig.10 Only some of the components related to the present disclosure in the electronic device 10 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 10 may also include any other appropriate components.

[0206] Exemplary computer program products and computer-readable storage media

[0207] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the training method of the motion state estimation model and the steps in the motion state estimation method according to various embodiments of the present application described in the above-mentioned "Exemplary Method" section of this specification.

[0208] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0209] In addition, an embodiment of the present application may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps in the training method of the motion state estimation model and the motion state estimation method according to various embodiments of the present application described in the above "Exemplary Method" section of this specification.

[0210] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0211] The basic principles of the present application are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present application are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present application. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, not for limitation, and the above details do not limit the present application to being implemented by adopting the above specific details.

[0212] The block diagrams of the devices, apparatuses, equipment, and systems involved in this application are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.

[0213] It should also be noted that in the apparatus, device and method of the present application, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0214] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0215] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A training method for a motion state estimation model, wherein the motion state estimation model comprises a feature extraction network and a motion state prediction network, and the method comprises: Acquire a sample image set, the sample image set comprising a first sample image taken at a first moment and a second sample image taken at a second moment, the first moment being earlier than the second moment; Using the feature extraction network, based on the first sample image and the second sample image, obtain a first image feature corresponding to the first sample image and a second image feature corresponding to the second sample image; Using the first image feature and the second image feature, perform feature transformation and feature fusion to obtain a first fusion feature in a target viewing angle space; Using the motion state prediction network to predict the first fusion feature, obtain a predicted motion state flow of the target object in the sample image set; Determining a model loss based on the predicted motion state flow and the reference motion state flow; The reference motion state flow is a tensor in the target viewing angle space used to characterize the true value of the motion state of the target object; Based on the model loss, training the motion state estimation model; Among them, the first image feature and the second image feature are used to perform feature transformation and feature fusion to obtain a first fused feature in the target viewing angle space, including: using a first transfer matrix to project the first image feature and the second image feature into the target viewing angle space to obtain a first projection feature corresponding to the first image feature in the target viewing angle space and a second projection feature corresponding to the second image feature in the target viewing angle space; using a second transfer matrix to transform the first projection feature to obtain a third projection feature corresponding to the first projection feature at the second moment; performing feature fusion of the third projection feature with the second projection feature to obtain the first fused feature; the first transfer matrix is ​​a homography matrix from the target viewing angle coordinate system to the image coordinate system; the second transfer matrix is ​​a vehicle posture change matrix from the first moment to the second moment.

2. The method according to claim 1, wherein: The feature extraction network includes a first feature extraction network and a second feature extraction network, and using the feature extraction network to obtain a first image feature corresponding to the first sample image and a second image feature corresponding to the second sample image based on the first sample image and the second sample image includes: Using the first feature extraction network, based on the first sample image, obtain the first image feature; The second image feature is obtained based on the second sample image using the second feature extraction network.

3. The method according to claim 1, wherein: The reference motion state flow is determined by the following steps: Acquire a point cloud data set, the point cloud data set comprising first point cloud data collected at the first moment and second point cloud data collected at the second moment; Determine, by using a target detection algorithm, first position information of the target object in the first point cloud data and second position information of the target object in the second point cloud data; Determine the change information of the orientation of the target object from the first moment to the second moment by using the first orientation information and the second orientation information; The initial motion state flow is updated using the orientation change information to obtain the reference motion state flow.

4. The method according to claim 3, wherein: The using the first position information and the second position information to determine the position change information of the target object from the first moment to the second moment includes: Using a third transfer matrix, projecting the first orientation information and the second orientation information into a target viewing angle space, obtaining first projection orientation information corresponding to the first orientation information in the target viewing angle space and second projection orientation information corresponding to the second orientation information in the target viewing angle space, wherein the third transfer matrix is ​​an external parameter matrix from a vehicle-mounted radar coordinate system to the target viewing angle coordinate system; Using the second transfer matrix, transforming the first projection orientation information to obtain third projection orientation information corresponding to the first projection orientation information at the second moment; The orientation change information is determined using the second projection orientation information and the third projection orientation information.

5. The method according to claim 3, wherein: The sample image set includes a plurality of the target objects, and determining the position change information of the target object from the first moment to the second moment by using the first position information and the second position information includes: Determine the corresponding relationship between the first position information and the second position information of each of the target objects in the plurality of target objects by using a target tracking algorithm; Based on the corresponding relationship, the first orientation information and the second orientation information of each target object are respectively used to determine the orientation change information of each target object from the first moment to the second moment.

6. A motion state estimation method, comprising: Acquire a set of images to be processed, the set of images to be processed comprising a first image to be processed taken at a third moment and a second image to be processed taken at a current moment, the third moment being earlier than the current moment; The motion state estimation model trained by the motion state estimation model training method described in any one of claims 1 to 5 obtains the motion state flow of the target object at the current moment based on the image set to be processed.

7. A training device for a motion state estimation model, the motion state estimation model comprising a feature extraction network and a motion state prediction network, the device comprising: A sample acquisition module, configured to acquire a sample image set, wherein the sample image set includes a first sample image taken at a first moment and a second sample image taken at a second moment, wherein the first moment is earlier than the second moment; a feature extraction module, configured to obtain, by using the feature extraction network, a first image feature corresponding to the first sample image and a second image feature corresponding to the second sample image based on the first sample image and the second sample image; A feature fusion module, used to perform feature transformation and feature fusion using the first image feature and the second image feature to obtain a first fusion feature in a target viewing angle space; A prediction module, configured to predict the first fusion feature using the motion state prediction network to obtain a predicted motion state flow of the target object in the sample image set; A loss determination module, configured to determine a model loss based on the predicted motion state flow and the reference motion state flow; The reference motion state flow is a tensor in the target viewing angle space used to characterize the true value of the motion state of the target object; A training module, used for training the motion state estimation model based on the model loss; The feature fusion module is specifically used to project the first image feature and the second image feature into the target viewing angle space using a first transfer matrix to obtain a first projection feature corresponding to the first image feature in the target viewing angle space and a second projection feature corresponding to the second image feature in the target viewing angle space; transform the first projection feature using a second transfer matrix to obtain a third projection feature corresponding to the first projection feature at the second moment; The third projection feature is fused with the second projection feature to obtain the first fused feature; the first transfer matrix is ​​a homography matrix from a target view coordinate system to an image coordinate system; and the second transfer matrix is ​​a vehicle posture change matrix from a first moment to a second moment.

8. A motion state estimation device, comprising: An image acquisition module, used to acquire a set of images to be processed, wherein the set of images to be processed includes a first image to be processed taken at a third moment and a second image to be processed taken at a current moment, wherein the third moment is earlier than the current moment; A motion state estimation module is used to obtain a motion state flow of a target object at a current moment based on the set of images to be processed by using a motion state estimation model trained using the motion state estimation model training method described in any one of claims 1 to 5.

9. A computer-readable storage medium storing a computer program, wherein the computer program is used to execute the training method of the motion state estimation model described in any one of claims 1 to 5 or the motion state estimation method described in claim 6.

10. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the training method of the motion state estimation model described in any one of claims 1-5 or the motion state estimation method described in claim 6.

Images