Prediction method and device for motion state corresponding to space grid and automatic driving vehicle

Abstract

The invention provides a prediction method and device for a motion state corresponding to a space grid and an automatic driving vehicle, and relates to the field of automatic driving, in particular to the technical fields of image processing, environmental perception and the like. According to the specific implementation scheme, the method comprises the steps of determining feature vectors of a plurality of space grids in a first space based on a plurality of images of the first space collected at the current moment; and performing multiple times of iteration processing on the feature vectors of the plurality of space grids in the first space and the motion state prediction information corresponding to the plurality of space grids in the second space to obtain the motion state prediction information corresponding to the plurality of space grids in the first space, the motion state prediction information corresponding to the plurality of space grids in the second space is obtained at a historical moment.

Description

Technical Field

[0001] The present disclosure relates to the field of autonomous driving, and in particular to technical fields such as image processing and environmental perception. Background Art

[0002] In the field of autonomous driving, the accuracy of the autonomous driving vehicle's perception of surrounding obstacles plays a vital role in the safety of the autonomous driving vehicle. In the existing technology, autonomous driving vehicles usually use point cloud data collected by radar to perceive or predict obstacles in the surrounding space, thereby determining the relevant features or related information of the surrounding space. However, this method requires the use of laser radar, which leads to high costs. Therefore, how to reduce costs while ensuring that the autonomous driving vehicle accurately predicts the relevant features or related information in the surrounding space has become a technical problem to be solved. Summary of the invention

[0003] The present disclosure provides a method, device, equipment, storage medium and autonomous driving vehicle for predicting the motion state corresponding to a spatial grid.

[0004] According to one aspect of the present disclosure, a method for predicting a motion state corresponding to a spatial grid is provided, comprising:

[0005] Determine, based on a plurality of images of the first space collected at a current moment, feature vectors of a plurality of spatial grids in the first space;

[0006] The feature vectors of the multiple spatial grids in the first space and the motion state prediction information corresponding to the multiple spatial grids in the second space are iterated multiple times to obtain the motion state prediction information corresponding to the multiple spatial grids in the first space, wherein the motion state prediction information corresponding to the multiple spatial grids in the second space is obtained at a historical moment.

[0007] According to one aspect of the present disclosure, a device for predicting a motion state corresponding to a spatial grid is provided, comprising:

[0008] A feature extraction module, used to determine feature vectors of a plurality of spatial grids in the first space based on a plurality of images of the first space collected at a current moment;

[0009] A prediction module is used to perform multiple iterative processing on the feature vectors of multiple spatial grids in the first space and the motion state prediction information corresponding to multiple spatial grids in the second space to obtain the motion state prediction information corresponding to the multiple spatial grids in the first space, wherein the motion state prediction information corresponding to the multiple spatial grids in the second space is obtained at a historical moment.

[0010] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0011] at least one processor; and

[0012] a memory communicatively connected to the at least one processor; wherein,

[0013] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0014] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0015] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0016] According to another aspect of the present disclosure, an autonomous driving vehicle is provided, comprising the above-mentioned electronic device.

[0017] By adopting the method provided in this embodiment, the motion states of multiple spatial grids in the first space at the current moment are predicted through multiple images at the current moment and motion state prediction information at historical moments. In this way, the motion states of each spatial grid in the current space can be predicted by using only the images collected at the current moment and combining them with the motion state prediction information at historical moments. Since the prediction results at historical moments are combined in the prediction process, the accuracy of the prediction of the motion states of the surrounding spatial grids at the current moment can be guaranteed. In addition, since only the images at the current moment need to be collected to realize the prediction of the motion states of each spatial grid in the current space, the cost is reduced.

[0018] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0020] Figure 1 is a flow chart of a method for predicting a motion state corresponding to a spatial grid according to an embodiment of the present disclosure;

[0021] Figure 2 is a flow chart of a method for predicting a motion state corresponding to a spatial grid according to another embodiment of the present disclosure;

[0022] Figure 3 is a schematic block diagram of a device for predicting a motion state corresponding to a spatial grid according to an embodiment of the present disclosure;

[0023] Figure 4 is a schematic block diagram of a device for predicting a motion state corresponding to a spatial grid according to another embodiment of the present disclosure;

[0024] Figure 5 is a block diagram of an electronic device for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0026] Figure 1 It is a schematic flow chart of a method for predicting a motion state corresponding to a spatial grid provided according to an embodiment of the present disclosure, comprising:

[0027] S110, determining feature vectors of a plurality of spatial grids in the first space based on a plurality of images of the first space collected at a current moment.

[0028] S120, performing multiple iterative processing on the feature vectors of the multiple spatial grids in the first space and the motion state prediction information corresponding to the multiple spatial grids in the second space to obtain the motion state prediction information corresponding to the multiple spatial grids in the first space, wherein the motion state prediction information corresponding to the multiple spatial grids in the second space is obtained at a historical moment.

[0029] The above-mentioned method for predicting the motion state corresponding to the spatial grid can be implemented by an electronic device. The electronic device can be a terminal or a server. Exemplarily, the electronic device can be a vehicle-mounted terminal, or other terminal device that has computing capabilities and can communicate with the vehicle. Exemplarily, the server can be a single server that can communicate with the vehicle, or one or more servers in a server cluster. It should be understood that the above is only an exemplary description of the execution subject that can execute the method for predicting the motion state corresponding to the spatial grid provided in this embodiment. The actual processing may not be limited to the device types mentioned in the above examples. As long as the electronic device can execute the method for predicting the motion state corresponding to the spatial grid provided in this embodiment, it is within the protection scope of this embodiment.

[0030] By adopting the method provided in this embodiment, the motion states of multiple spatial grids in the first space at the current moment are predicted through multiple images at the current moment and motion state prediction information at historical moments. In this way, the motion states of each spatial grid in the current space can be predicted by using only the images collected at the current moment and combining them with the motion state prediction information at historical moments. Since the prediction results at historical moments are combined in the prediction process, the accuracy of the prediction of the motion states of the surrounding spatial grids at the current moment can be guaranteed. In addition, since only the images at the current moment need to be collected to realize the prediction of the motion states of each spatial grid in the current space, the cost is reduced.

[0031] In an embodiment, the method of acquiring the multiple images of the first space captured at the current moment may include: acquiring the multiple images of the first space captured by the image sensor at the current moment.

[0032] It should be noted that the electronic device side that executes the prediction method of the motion state corresponding to the spatial grid of the present application can pre-set the collection interval between the collection moments. The specific duration of the collection interval can be configured according to actual conditions. For example, it can be 0.1 second, or 1 second, or longer or shorter, which is not limited here.

[0033] The current moment may refer to the current acquisition moment, that is, the electronic device may control the image sensor to acquire, and control the image sensor to perform acquisition once every acquisition interval. Specifically, obtaining multiple images of the first space acquired by the image sensor at the current moment may refer to the electronic device controlling the image sensor to acquire at the current moment, and acquiring multiple images of the first space acquired by the image sensor at the current moment.

[0034] The first space may refer to the space where the vehicle is currently located, or the space where the vehicle is located at the current moment.

[0035] The range of the first space may be related to a preset acquisition range and a perception range of an image sensor. The preset acquisition range may be configured according to actual conditions. For example, the preset acquisition range may be: with the image sensor as the origin, the range is plus or minus 120 meters longitudinally, plus or minus 120 meters horizontally, and plus or minus 3 meters vertically.

[0036] The image sensor may be any device or sensor provided on the vehicle and having an image acquisition function.

[0037] The number of image sensors may be one or more.

[0038] Exemplarily, the number of image sensors may be multiple, and different image sensors in the multiple image sensors may have different acquisition angles. For example, the multiple image sensors may include multiple cameras or multiple cameras arranged on the vehicle, and different cameras or different cameras have different acquisition angles.

[0039] In this case, the multiple images of the first space may be two-dimensional images of at least a partial range in the first space captured by each of the multiple image sensors. Since different image sensors among the multiple image sensors may have different acquisition angles, each image sensor may only capture a two-dimensional image within a partial range of the first space where the vehicle is currently located, different image sensors may capture two-dimensional images within different ranges within the first space where the vehicle is currently located, and the two-dimensional images captured by different image sensors may have partially overlapping ranges, but all image sensors may capture all two-dimensional images of the entire range within the first space where the vehicle is currently located, that is, the collection of two-dimensional images captured by all image sensors can cover the entire range of the first space. How all image sensors can specifically capture all two-dimensional images covering the entire range of the first space is related to the setting of the acquisition angle of each image sensor and is not limited here.

[0040] Exemplarily, the number of image sensors may be one, for example, the image sensor may include a surround view camera disposed on the vehicle. In this case, the multiple images of the first space may be multiple two-dimensional images in the surround view of the first space captured by the image sensor. It should be understood that each two-dimensional image may be an image of a partial range of the first space where the vehicle is located, and different two-dimensional images contain images of different ranges of the first space, but all two-dimensional images contain the entire range of the first space.

[0041] In one embodiment, the feature vectors of multiple spatial grids in the first space are determined based on multiple images of the first space collected at the current moment, including: extracting features from each pixel in each of the multiple images of the first space collected at the current moment to obtain a feature vector for each pixel in each image; and obtaining a feature vector for each of the multiple spatial grids in the first space based on the feature vector of each pixel in each image. The feature vector may be a semantic vector, which is used to describe shape, texture, color, etc., which are not listed here one by one.

[0042] The step of extracting features from each pixel in each of the multiple images of the first space collected at the current moment to obtain a feature vector for each pixel in each image may include: extracting features from each pixel in each of the multiple images of the first space collected at the current moment through a feature extraction network to obtain a feature vector for each pixel in each image. The feature extraction network may be a ResNet (Residual Network), and specifically, ResNet may include one of the following: ResNet50, Resnet101.

[0043] For example, performing feature extraction on each pixel in each of the multiple images of the first space collected at the current moment through a feature extraction network to obtain a feature vector for each pixel in each image can include: inputting the mth image of the first space collected at the current moment into ResNet, and obtaining a feature vector for each pixel in the mth image output by ResNet, wherein m is an integer greater than or equal to 1, and the mth image is any one of the multiple images. Since the processing of each image is the same as that of the mth image, they will not be described one by one.

[0044] The method of obtaining the feature vector of each spatial grid in the first space based on the feature vector of each pixel in each image may include: generating multiple spatial grids in the first space based on the first spatial range and spatial grid parameters; mapping the feature vector of each pixel in each image to the corresponding spatial grid in the first space to obtain the feature vector of one or more pixels corresponding to each of the multiple spatial grids in the first space; and obtaining the feature vector of each spatial grid in the first space based on the feature vector of one or more pixels corresponding to each spatial grid in the first space.

[0045] The spatial grid parameter may include at least one of the following: the size of the spatial grid and the number of spatial grids. The size of the spatial grid and the number of spatial grids may be set according to actual conditions and are not limited in this application. The size of the spatial grid may also be referred to as the resolution of the spatial grid.

[0046] The step of mapping the feature vector of each pixel in each image to the corresponding spatial grid in the first space to obtain the feature vector of one or more pixels corresponding to each of the multiple spatial grids in the first space may include: determining the spatial grid corresponding to each pixel in each image in the first space based on the correspondence between the coordinates of each image and the coordinates of the first space; mapping the feature vector of each pixel in each image to the spatial grid corresponding to each pixel in the first space to obtain the feature vector of one or more pixels corresponding to each of the multiple spatial grids in the first space. The coordinates of each image may be the coordinates in the image coordinate system, and the specific setting method of the image coordinate system is not limited in this embodiment; the coordinates of the first space may be the coordinates in the spatial coordinate system, and the spatial coordinate system may be a system with the vehicle as the origin, the horizontal plane as the xy plane, and the vertical line of the horizontal plane as the z axis. It should be noted that this is only an exemplary description of the spatial coordinate system. In actual processing, the spatial coordinate system may also be set in other ways, which are not limited or exhaustive here. The correspondence between the coordinates of each image and the coordinates of the first space may refer to the correspondence between the coordinates of each image in the image coordinate system and the coordinates of the first space in the space coordinate system. The generation and specific content of the correspondence are not limited in this embodiment.

[0047] Taking the zth (z is a positive integer) pixel in the kth (k is a positive integer) image among multiple images as an example, mapping the feature vector of each pixel in each image to the spatial grid corresponding to each pixel in the first space to obtain the feature vector of one or more pixels corresponding to each of the multiple spatial grids in the first space may include: mapping the feature vector of the zth pixel in the kth image to the nth spatial grid corresponding to the zth pixel in the first space to obtain the feature vector of a pixel corresponding to the nth spatial grid, where n is a positive integer.

[0048] The obtaining of the feature vector of each of the multiple spatial grids in the first space based on the feature vector of one or more pixel points corresponding to each spatial grid in the first space may include: calculating the feature vector of one or more pixel points corresponding to each spatial grid in the first space based on a specified calculation method to obtain the feature vector of each of the multiple spatial grids in the first space. The specified calculation method may be one of the following: average value calculation, maximum value calculation, etc. It should be understood that this is only an exemplary description. In actual processing, the specified calculation method may also include other related statistical methods, which are not limited or exhaustive here.

[0049] In one example, based on the first spatial range and spatial grid parameters, multiple spatial grids in the first space are generated; the feature vector of each pixel in each image is mapped to the corresponding spatial grid in the first space to obtain the feature vector of one or more pixels corresponding to each of the multiple spatial grids in the first space; based on the feature vector of one or more pixels corresponding to each spatial grid in the first space, the feature vector of each of the multiple spatial grids in the first space is obtained, which can be implemented by a spatial generation network. Specifically, the feature vector of each pixel in each image can be input into the spatial generation network to obtain the feature vector of each of the multiple spatial grids in the first space output by the spatial generation network. Among them, the spatial generation network can be a Cam2voxel (Camera 2voxel, camera voxel) network.

[0050] In one example, the feature vector of each pixel in each image is mapped to the corresponding spatial grid in the first space to obtain the feature vector of one or more pixels corresponding to each of the multiple spatial grids in the first space, which can be implemented by a first mapping model. Specifically, the feature vector of each pixel in each image can be input into the first mapping model to obtain the feature vector of one or more pixels corresponding to each of the multiple spatial grids in the first space output by the first mapping model. The first mapping model can be obtained by training a transformer model, and the present application does not limit the specific training method of the transformer model.

[0051] In one example, the feature vector of each pixel in each image is mapped to the corresponding spatial grid in the first space to obtain the feature vector of one or more pixels corresponding to each of the multiple spatial grids in the first space; based on the feature vector of one or more pixels corresponding to each spatial grid in the first space, the feature vector of each of the multiple spatial grids in the first space is obtained, which can be implemented by a second mapping model. Specifically, the feature vector of each pixel in each image can be input into the second mapping model to obtain the feature vector of each of the multiple spatial grids in the first space output by the second mapping model. The second mapping model can be obtained by training the converter model, and the present application does not limit the specific training method of the converter model.

[0052] In one embodiment, the feature vectors of the multiple spatial grids in the first space and the motion state prediction information corresponding to the multiple spatial grids in the second space are subjected to multiple iterative processing to obtain the motion state prediction information corresponding to the multiple spatial grids in the first space, including: in the i-th iterative processing of the multiple iterative processing, based on the feature vectors of the multiple spatial grids in the first space and the motion state prediction information corresponding to the multiple spatial grids in the second space, generating the information to be processed for the i-th iterative processing, wherein i is a positive integer; inputting the information to be processed for the i-th iterative processing and the i-th first conditional vector into the target model to obtain the i-th motion state prediction information corresponding to the multiple spatial grids in the first space output by the target model; when the i-th iterative processing satisfies the constraint conditions, using the i-th motion state prediction information corresponding to the multiple spatial grids in the first space as the motion state prediction information corresponding to the multiple spatial grids in the first space.

[0053] Among them, the motion state includes at least one of the following: speed, direction, acceleration, etc. Correspondingly, the i-th motion state prediction information includes at least one of the following: the i-th predicted speed, the i-th predicted direction, the i-th predicted acceleration, etc., which are not listed one by one here.

[0054] The i-th motion state prediction information corresponding to any spatial grid in the first space can be used to represent at least one of the i-th predicted speed, the i-th predicted direction, the i-th predicted acceleration, etc. of the occupied object corresponding to the spatial grid in the first space. The occupied object may include one of the following: an obstacle vehicle, a pedestrian, a green belt, a building, etc.

[0055] It should be understood that the occupied objects corresponding to different spatial grids in the first space may be the same or different. There is no limitation here on whether different spatial grids correspond to the same occupied object. For each spatial grid, at least one of the i-th predicted speed, i-th predicted direction, i-th predicted acceleration, etc. of the corresponding occupied object can be determined independently.

[0056] The number of the second space may be one or more.

[0057] If the number of the second space is one, the second space may refer to the space where the vehicle is located at a historical moment, and the historical moment may refer to the last collection moment adjacent to the current moment. The second space may be the same as or at least partially different from the first space. Specifically, the multiple spatial grids of the second space may be the same as the multiple spatial grids of the first space; or, the multiple spatial grids of the second space may be partially the same as or at least partially different from the multiple spatial grids of the first space.

[0058] If there are multiple second spaces, each of the multiple second spaces refers to the space where the vehicle is located at each historical moment, wherein different second spaces correspond to different historical moments, that is, multiple second spaces correspond to multiple historical moments, the last historical moment among the multiple historical moments is the previous collection moment adjacent to the current moment, and the interval between two adjacent historical moments among the multiple historical moments is equal to the collection interval, and the relevant description of the collection interval is the same as that in the previous embodiment and will not be repeated.

[0059] Among the multiple second spaces, different second spaces may be the same or at least partially different. Specifically, multiple space grids of different second spaces may be the same; or, multiple space grids of different second spaces may be partially the same and at least partially different. Furthermore, any second space among the multiple second spaces may be the same or at least partially different from the first space. Specifically, multiple space grids of any second space may be the same as multiple space grids of the first space; or, multiple space grids of any second space may be partially the same as multiple space grids of the first space.

[0060] The motion state prediction information corresponding to any spatial grid in any second space can be used to represent at least one of the predicted speed, predicted direction, predicted acceleration, etc. of the occupied object corresponding to the spatial grid in the second space.

[0061] Optionally, the generating of the information to be processed for the i-th iterative processing based on the feature vectors of the multiple spatial grids in the first space and the motion state prediction information corresponding to the multiple spatial grids in the second space may include one of the following: using the feature vectors of the multiple spatial grids in the first space and the motion state prediction information corresponding to the multiple spatial grids in the second space as the information to be processed for the i-th iterative processing; using the feature vectors of the multiple spatial grids in the first space, the motion state prediction information corresponding to the multiple spatial grids in the second space, and the feature vectors of the multiple spatial grids in the second space as the information to be processed for the i-th iterative processing.

[0062] The target model may be obtained by training a preset model, wherein the preset model may be a diffusion model. In this embodiment, the target model is used to denoise the content to be processed, and the denoising process may also be called a reverse denoising process, which may make the content to be processed clearer or more accurate, and then predict the prediction information related to the content to be processed based on the clearer or more accurate content to be processed.

[0063] The step of inputting the information to be processed and the i-th first conditional vector processed by the i-th iteration into the target model to obtain the i-th motion state prediction information corresponding to multiple spatial grids in the first space output by the target model may include: inputting the information to be processed and the i-th first conditional vector processed by the i-th iteration into the target model to obtain the i-th prediction information corresponding to multiple spatial grids in the first space output by the target model, wherein the i-th prediction information corresponding to the multiple spatial grids in the first space includes the i-th motion state prediction information corresponding to the multiple spatial grids in the first space.

[0064] Wherein, when i is equal to 1, the i-th first conditional vector is a random noise vector; when i is greater than 1, the i-th first conditional vector is the i-1th motion state prediction information corresponding to multiple spatial grids in the first space. The random noise vector can be obtained by encoding according to Gaussian operation, and the specific method of obtaining is not limited in this application.

[0065] Optionally, the generating of the information to be processed for the i-th iterative processing based on the feature vectors of the multiple spatial grids in the first space and the motion state prediction information corresponding to the multiple spatial grids in the second space may include one of the following: using the feature vectors of the multiple spatial grids in the first space, the motion state prediction information corresponding to the multiple spatial grids in the second space, and the occupancy state prediction information corresponding to the multiple spatial grids in the second space as the information to be processed for the i-th iterative processing; using the feature vectors of the multiple spatial grids in the first space, the motion state prediction information corresponding to the multiple spatial grids in the second space, the occupancy state prediction information corresponding to the multiple spatial grids in the second space, and the feature vectors of the multiple spatial grids in the second space as the information to be processed for the i-th iterative processing.

[0066] The step of inputting the information to be processed of the i-th iteration and the i-th first conditional vector into the target model to obtain the i-th motion state prediction information corresponding to the multiple spatial grids in the first space output by the target model may include: inputting the information to be processed of the i-th iteration and the i-th first conditional vector into the target model to obtain the i-th prediction information corresponding to the multiple spatial grids in the first space output by the target model, wherein the i-th prediction information corresponding to the multiple spatial grids in the first space includes the i-th motion state prediction information corresponding to the multiple spatial grids in the first space and the i-th occupancy state prediction information corresponding to the multiple spatial grids in the first space.

[0067] Among them, when i is equal to 1, the i-th first conditional vector is a random noise vector; when i is greater than 1, the i-th first conditional vector is the i-1th motion state prediction information corresponding to multiple spatial grids in the first space and the i-1th occupancy state prediction information corresponding to multiple spatial grids in the first space.

[0068] Taking the g-th spatial grid (g is a positive integer) among the multiple spatial grids of the first space as an example, the i-th occupancy state prediction information corresponding to the multiple spatial grids in the first space may include: the i-th occupancy prediction information corresponding to the g-th spatial grid in the first space.

[0069] In the case where the prediction information of whether the i-th occupancy corresponding to the g-th spatial grid in the first space is occupied, the prediction information of the i-th occupancy state corresponding to the g-th spatial grid in the first space may further include: an occupancy attribute corresponding to the g-th spatial grid in the first space. The occupancy attribute includes at least one of the following: texture, shape, color, occupied object, etc., which are not listed here one by one.

[0070] When the i-th iterative processing satisfies the constraint conditions, the i-th motion state prediction information corresponding to the multiple spatial grids in the first space is used as the motion state prediction information corresponding to the multiple spatial grids in the first space, including: judging whether the i-th iterative processing satisfies the constraint conditions, and when the i-th iterative processing satisfies the constraint conditions, the i-th motion state prediction information corresponding to the multiple spatial grids in the first space is used as the motion state prediction information corresponding to the multiple spatial grids in the first space.

[0071] The method of determining whether the i-th iterative processing satisfies the constraint conditions includes at least one of the following: determining that the i-th iterative processing satisfies the constraint conditions when the i-th iterative processing is the last iterative processing; determining that the i-th iterative processing satisfies the constraint conditions when the i-th occupancy state prediction information corresponding to multiple spatial grids in the first space meets the quality requirements.

[0072] Among them, whether the i-th iteration process is the last iteration process can be determined according to a preset number of iterations. For example, if the number of i-th iteration processes reaches the preset number of iterations, the i-th iteration process can be determined to be the last iteration process. Otherwise, it is determined that the i-th iteration process is not the last iteration process. The preset number of iterations can be configured according to actual conditions, for example, it can be 10 times, or 5 times, or larger or smaller, and this embodiment does not limit it.

[0073] The method of determining whether the i-th occupancy state prediction information corresponding to the plurality of spatial grids in the first space meets the quality requirement includes: determining whether the i-th occupancy state prediction information corresponding to the plurality of spatial grids in the first space meets the quality requirement when relevant parameters of the i-th occupancy state prediction information corresponding to the plurality of spatial grids in the first space meet the preset conditions. The relevant parameters include at least one of the following: mean square error, peak signal-to-noise ratio, brightness, contrast, etc.

[0074] In addition, the method may further include: if the i-th iteration process does not satisfy the constraint condition, executing the i+1-th iteration process of the multiple iteration processes. The i+1-th iteration process of the multiple iteration processes is the same as the i-th iteration process of the multiple iteration processes, and will not be described in detail herein.

[0075] It should be noted that after obtaining the motion state prediction information corresponding to the multiple spatial grids in the first space, the motion state prediction information corresponding to each spatial grid in the first space can be directly used for prediction processing at the next moment.

[0076] Alternatively, after obtaining the motion state prediction information corresponding to multiple spatial grids in the first space, the following processing can also be performed: based on the occupied objects corresponding to each spatial grid in the multiple spatial grids in the first space, determine that each occupied object in the first space corresponds to one or more spatial grids, based on the motion state prediction information corresponding to each spatial grid corresponding to each occupied object, obtain the motion state prediction information of each occupied object, and based on the motion state prediction information corresponding to each spatial grid corresponding to each occupied object, update the motion state prediction information corresponding to each spatial grid in the first space.

[0077] Taking any occupied object as an example, based on the motion state prediction information corresponding to each spatial grid corresponding to each object, the motion state prediction information of the object is obtained, which may include at least one of the following: based on the predicted speed in the motion state prediction information corresponding to each spatial grid corresponding to the occupied object, the predicted speed of the occupied object is obtained; based on the predicted direction in the motion state prediction information corresponding to each spatial grid corresponding to the occupied object, the predicted direction of the occupied object is obtained; based on the predicted acceleration in the motion state prediction information corresponding to each spatial grid corresponding to the occupied object, the predicted acceleration of the occupied object is obtained. Among them, the specific method of obtaining the predicted speed, predicted direction, and predicted acceleration of the occupied object can be configured according to the actual situation, such as calculating the average value, taking the maximum value, taking the minimum value, etc., which is not limited or exhaustive here.

[0078] Still taking any occupied object as an example, the motion state prediction information corresponding to each spatial grid in the first space is updated based on the motion state prediction information of each occupied object. It can be that the motion state prediction information corresponding to each spatial grid corresponding to the occupied object in the first space is set as the motion state prediction information of the occupied object.

[0079] In this way, the target model can be used to perform multiple iterative processing on the feature vectors of multiple spatial grids in the first space at the current moment and the motion state prediction information of multiple spatial grids in the second space obtained at historical moments, and finally obtain the motion state prediction information of multiple spatial grids in the first space at the current moment. In this way, by performing multiple iterative processing methods through the target model, the accuracy and efficiency of the motion state prediction information corresponding to the multiple spatial grids in the first space can be improved. Moreover, in each iterative processing, the previous output result is referred to, so that the accuracy of the motion state prediction information corresponding to the multiple spatial grids in the first space can be further improved.

[0080] In one embodiment, when there is no historical moment (i.e., the first collection moment in the current driving task of the target vehicle), the method further includes: using the initial speed as the motion state prediction information of the multiple spatial grids in the first space at the current moment. The initial speed can be set according to actual conditions, for example, the initial speed can be 0 or set to null.

[0081] Combination Figure 2 The above-mentioned implementation modes are exemplarily described, including:

[0082] S201, acquiring a plurality of images of a first space captured by an image sensor at a current moment (ie, a plurality of two-dimensional images in a surround view of the first space captured by a surround view camera).

[0083] S202, performing feature extraction on each pixel point in each image of a plurality of images in the first space collected at the current moment, to obtain a feature vector of each pixel point in each image.

[0084] S203, input the feature vector of each pixel point in each image of the multiple images in the first space into the space generation network (ie, Cam2voxel), and obtain the feature vector of each spatial grid in the multiple spatial grids in the first space output by the space generation network.

[0085] S204, performing multiple iterative processing on the feature vectors of the multiple spatial grids in the first space and the motion state prediction information corresponding to the multiple spatial grids in the second space to obtain the motion state prediction information corresponding to the multiple spatial grids in the first space.

[0086] The specific process of multiple iterative processing includes:

[0087] S2041, in the i-th iteration of the multiple iterations, based on the feature vectors of the multiple spatial grids in the first space and the motion state prediction information corresponding to the multiple spatial grids in the second space, generate the information to be processed for the i-th iteration; input the information to be processed for the i-th iteration and the i-th first conditional vector into the target model to obtain the i-th motion state prediction information corresponding to the multiple spatial grids in the first space output by the target model. Wherein, when i is equal to 1, the i-th first conditional vector is a random noise vector; when i is greater than 1, the i-th first conditional vector is the i-1-th motion state prediction information corresponding to the multiple spatial grids in the first space.

[0088] S2042, determine whether the i-th iterative processing satisfies the constraint conditions. If the constraint conditions are satisfied, use the i-th motion state prediction information corresponding to the multiple spatial grids in the first space as the motion state prediction information corresponding to the multiple spatial grids in the first space, and terminate the processing; if the constraint conditions are not satisfied, return to S2041 to execute the i+1-th iterative processing of the multiple iterative processing.

[0089] In one embodiment, the method further includes: training a preset model multiple times based on samples and sample labels to obtain the target model, wherein the samples include feature vectors of multiple spatial grids in a third space at a first moment, and motion state information corresponding to multiple spatial grids in a fourth space at a second moment, and the first moment is later than the second moment.

[0090] The preset model may be a diffusion model.

[0091] Among them, the method for obtaining the feature vectors of the multiple spatial grids in the third space at the first moment is similar to the method for obtaining the feature vectors of the multiple spatial grids in the first space mentioned above, and will not be repeated here. In addition, the feature vectors of the multiple spatial grids in the third space at the first moment can also be obtained by other methods, which are not limited in this application.

[0092] The first moment may refer to a first collection moment, wherein the relevant description of the collection moment is the same as that in the above implementation manner and is not repeated here.

[0093] The third space may refer to the space where the vehicle is located at the first moment, or the space where the vehicle is located at the first moment. The range of the third space may be related to the preset acquisition range and the perception range of the image sensor. The preset acquisition range can be configured according to actual conditions, and the specific configuration method can be set according to actual conditions, which is not limited in this application.

[0094] The second time point may be one or more.

[0095] If the second moment is one, the second moment refers to the previous collection moment adjacent to the first moment, and the corresponding fourth space at the second moment may refer to the space where the vehicle is located at the second moment. The fourth space may be the same as or at least partially different from the third space. Specifically, the multiple spatial grids of the fourth space may be the same as the multiple spatial grids of the third space; or, the multiple spatial grids of the fourth space may be partially the same as or at least partially different from the multiple spatial grids of the third space.

[0096] If there are multiple second moments, the second moment refers to each historical collection moment before the first moment, and the corresponding fourth space at the second moment may refer to the space where the vehicle is located at each historical collection moment, wherein different fourth spaces correspond to different historical moments, that is, multiple fourth spaces correspond to multiple historical moments, the last historical moment among the multiple historical moments is the previous collection moment adjacent to the first moment, and the interval between two adjacent historical moments among the multiple historical moments is equal to the collection interval, and the relevant description of the collection interval is the same as that in the previous embodiment and will not be repeated.

[0097] Among the multiple fourth spaces, different fourth spaces may be the same or at least partially different. Specifically, the multiple space grids of different fourth spaces may be the same; or, the multiple space grids of different fourth spaces may be partially the same and at least partially different. Moreover, any fourth space among the multiple fourth spaces may be the same or at least partially different from the third space. Specifically, the multiple space grids of any fourth space may be the same as the multiple space grids of the third space; or, the multiple space grids of any fourth space may be partially the same as the multiple space grids of the third space.

[0098] The method for obtaining the motion state information corresponding to the multiple spatial grids in the fourth space at the second moment is not limited in this application. For example, the motion state information corresponding to the multiple spatial grids in the fourth space at the second moment may be obtained based on the point cloud data collected by the point cloud collection device on the vehicle at the second moment, or detected by other speed detection devices, which are not limited or exhaustive here. The point cloud collection device may be one of the following: a laser radar, a millimeter wave radar, and the like.

[0099] In this way, the preset model is iteratively trained multiple times based on the feature vectors of multiple spatial grids in the third space at the first moment and the motion state information corresponding to multiple spatial grids in the fourth space at the second moment contained in the sample, and the sample label to obtain the target model. In this way, the preset model can be trained multiple times so that the trained preset model can predict the motion state information corresponding to multiple spatial grids in the space according to the image information, so that the accuracy of the prediction of the motion state of the spatial grid by the trained target model can be guaranteed.

[0100] In one embodiment, the preset model is trained multiple times based on samples and sample labels to obtain the target model, including: in the jth training of the multiple trainings, the sample and the jth second conditional vector are input into the preset model, and the preset model outputs the jth motion state prediction information corresponding to the multiple spatial grids in the third space at the first moment, wherein j is a positive integer; based on the jth motion state prediction information and sample labels corresponding to the multiple spatial grids in the third space at the first moment, the jth loss is determined; based on the jth loss, the preset model is trained to obtain the preset model after the jth training; when the jth training is the last training among the multiple trainings, the preset model after the jth training is used as the target model.

[0101] Optionally, the step of inputting the sample and the j-th second conditional vector into a preset model to obtain the j-th motion state prediction information corresponding to the multiple spatial grids in the third space at the first moment output by the preset model includes: inputting the sample and the j-th second conditional vector into a preset model to obtain the j-th prediction information corresponding to the multiple spatial grids in the third space at the first moment output by the preset model, wherein the j-th prediction information corresponding to the multiple spatial grids in the third space at the first moment includes the j-th motion state prediction information corresponding to the multiple spatial grids in the third space at the first moment.

[0102] The samples include: feature vectors of a plurality of spatial grids in a third space at a first moment, and motion state information corresponding to a plurality of spatial grids in a fourth space at a second moment.

[0103] Alternatively, the sample includes: feature vectors of multiple spatial grids in a third space at a first moment, motion state information corresponding to multiple spatial grids in a fourth space at a second moment, and feature vectors of multiple spatial grids in the fourth space at the second moment. The method for obtaining the feature vectors of the multiple spatial grids in the fourth space at the second moment is not limited in this application.

[0104] The sample label may include: a motion state sample label. The motion state sample label may include: a real motion state corresponding to each of the multiple spatial grids in the third space at the first moment. The real motion state corresponding to each spatial grid may include at least one of the real position, real speed, and real direction corresponding to each spatial grid; the method for obtaining the real motion state corresponding to each spatial grid is not limited in this application.

[0105] When j is equal to 1, the j-th second conditional vector is obtained by adding a random noise coding vector to the sample label; when j is greater than 1, the j-th second conditional vector is obtained by adding a random noise vector to the j-1-th second conditional vector.

[0106] Adding a random noise coding vector to the sample label to obtain the j-th second conditional vector may include one of the following: adding a random noise coding vector to the real motion state corresponding to each spatial grid among multiple spatial grids within the third space of the motion state sample label to obtain the j-th second conditional vector; adding a random noise coding vector to the real motion state corresponding to some spatial grids among the multiple spatial grids within the third space of the motion state sample label to obtain the j-th second conditional vector.

[0107] Adding a random noise vector to the j-1th second condition vector to obtain the jth second condition vector is similar to the above process and will not be described in detail.

[0108] The determining of the jth loss based on the jth motion state prediction information and sample labels corresponding to the multiple spatial grids in the third space at the first moment includes: determining the jth loss based on the jth motion state prediction information and motion state sample labels corresponding to the multiple spatial grids in the third space at the first moment. The jth loss may be calculated using a cross entropy loss function, which is not limited in this embodiment.

[0109] Optionally, the step of inputting the sample and the j-th second conditional vector into a preset model to obtain the j-th motion state prediction information corresponding to the multiple spatial grids in the third space at the first moment output by the preset model includes: inputting the sample and the j-th second conditional vector into the preset model to obtain the j-th prediction information corresponding to the multiple spatial grids in the third space at the first moment output by the preset model, wherein the j-th prediction information corresponding to the multiple spatial grids in the third space at the first moment includes the j-th motion state prediction information corresponding to the multiple spatial grids in the third space at the first moment and the j-th occupancy state prediction information corresponding to the multiple spatial grids in the third space at the first moment.

[0110] The sample includes: feature vectors of multiple spatial grids in the third space at the first moment, motion state information corresponding to multiple spatial grids in the fourth space at the second moment, and occupancy state information corresponding to multiple spatial grids in the fourth space at the second moment. The acquisition method of the occupancy state information corresponding to the multiple spatial grids in the fourth space at the second moment is not limited in this application. For example, the occupancy state information corresponding to the multiple spatial grids in the fourth space at the second moment may be obtained based on the point cloud data collected by the point cloud collection device on the vehicle at the second moment, or detected by other occupancy state detection devices, which are not limited or exhaustive here.

[0111] Alternatively, the sample may include: feature vectors of multiple spatial grids in the third space at the first moment, motion state information corresponding to multiple spatial grids in the fourth space at the second moment, occupancy state information corresponding to multiple spatial grids in the fourth space at the second moment, and feature vectors of multiple spatial grids in the fourth space at the second moment.

[0112] The sample labels may include: motion state sample labels and occupancy state sample labels.

[0113] The motion state sample label may be the real motion state corresponding to each of the plurality of spatial grids in the third space. The acquisition method of the real motion state corresponding to each of the plurality of spatial grids in the third space is the same as the above example, which will not be repeated here.

[0114] The occupancy state sample label may be a real occupancy state corresponding to each of a plurality of space grids in the third space.

[0115] The method of obtaining the actual occupancy status corresponding to each of the multiple spatial grids in the third space may include: obtaining point cloud information in the third space; and obtaining the actual occupancy status corresponding to each of the multiple spatial grids in the third space based on the point cloud information in the third space.

[0116] The obtaining of the real occupancy state corresponding to each of the multiple spatial grids in the third space based on the point cloud information in the third space may include: mapping each point cloud feature in the point cloud information in the third space to the corresponding spatial grid in the third space, obtaining one or more point cloud features corresponding to each of the multiple spatial grids in the third space; obtaining the real occupancy state corresponding to each of the multiple spatial grids in the third space based on the one or more point cloud features corresponding to each spatial grid in the third space. Any point cloud feature can be used to represent geometric features, color features, texture features, etc., which are not exhaustive here.

[0117] The mapping of each point cloud feature in the point cloud information in the third space to the corresponding spatial grid in the third space to obtain one or more point cloud features corresponding to each of the multiple spatial grids in the third space may include: based on the correspondence between the coordinates of the point cloud information in the third space and the coordinates of the third space, mapping each point cloud feature in the point cloud information in the third space to the corresponding spatial grid in the third space to obtain one or more point cloud features corresponding to each of the multiple spatial grids in the third space. The coordinates of the point cloud information in the third space may be the coordinates in the radar coordinate system, and the specific setting method of the radar coordinate system is not limited in this embodiment; the method for determining the coordinates of the third space is the same as the method for determining the coordinates of the first space mentioned above, which will not be repeated here. The correspondence between the coordinates of the point cloud information in the third space and the coordinates of the third space may refer to the correspondence between the coordinates of the third space in the radar coordinate system and the coordinates of the third space in the spatial coordinate system, and the generation and specific content of the correspondence are not limited in this embodiment.

[0118] Taking the yth spatial grid (y is a positive integer) among the multiple spatial grids in the third space as an example, obtaining the actual occupancy state corresponding to each of the multiple spatial grids in the third space based on one or more point cloud features corresponding to each spatial grid in the third space may include: selecting the same and largest point cloud feature from the one or more point cloud features corresponding to the yth spatial grid in the third space as the actual occupancy state corresponding to the yth spatial grid.

[0119] The actual occupancy state can be called the true value of the semantic label. The actual occupancy state corresponding to any spatial grid can be represented by one-hot encoding (or one-bit effective encoding). For example, assuming that there are three optional occupancy states, and the actual occupancy state corresponding to a spatial grid is optional occupancy state 2, the one-hot encoding of the actual occupancy state corresponding to the spatial grid can be represented as {0,1,0}, where the first bit of the encoding corresponds to optional occupancy state 1, the second bit of the encoding corresponds to optional occupancy state 2, and the third bit of the encoding corresponds to optional occupancy state 3. If the second bit of the encoding is 1, it means that the actual occupancy state corresponding to the spatial grid is optional occupancy state 2.

[0120] When j is equal to 1, the j-th second conditional vector is obtained by adding a random noise coding vector to the sample label; when j is greater than 1, the j-th second conditional vector is obtained by adding a random noise vector to the j-1-th second conditional vector.

[0121] Adding a random noise coding vector to the sample label to obtain the j-th second condition vector may include: adding a random noise coding vector to the motion state sample label to obtain the j-th motion state condition vector; adding a random noise coding vector to the occupied state sample label to obtain the j-th occupied state condition vector; using the j-th motion state condition vector and the j-th occupied state condition vector as the j-th second condition vector. The specific processing method of adding a random noise coding vector to the motion state sample label to obtain the j-th motion state condition vector is similar to the above-mentioned embodiment and will not be repeated.

[0122] Adding a random noise coding vector to the occupancy state sample label to obtain the jth occupancy state condition vector may include one of the following: adding a random noise coding vector to the one-hot coding corresponding to each of the multiple spatial grids in the third space of the occupancy state sample label to obtain the jth occupancy state condition vector; adding a random noise coding vector to the one-hot coding corresponding to some of the multiple spatial grids in the third space of the occupancy state sample label to obtain the jth occupancy state condition vector. For example, assuming that there are 3 optional occupancy states, and the actual occupancy state corresponding to a certain spatial grid is optional occupancy state 2, the one-hot coding of the actual occupancy state corresponding to the spatial grid can be expressed as {0,1,0}. After adding the random noise coding vector to the one-hot coding, the following jth occupancy state condition vector {-0.5,0.6,0.1} may be obtained.

[0123] Adding a random noise vector to the j-1th second conditional vector to obtain the j-th second conditional vector includes: adding a random noise coding vector to the j-1th motion state conditional vector to obtain the j-th motion state conditional vector; adding a random noise coding vector to the j-1th occupancy state conditional vector to obtain the j-th occupancy state conditional vector; using the j-th motion state conditional vector and the j-th occupancy state conditional vector as the j-th second conditional vector. The process of adding a random noise coding vector to the j-1th motion state conditional vector to obtain the j-th motion state conditional vector is similar to the above example; the process of adding a random noise coding vector to the j-1th occupancy state conditional vector to obtain the j-th occupancy state conditional vector is also similar to the above example, and they are not repeated here.

[0124] The determining of the jth loss based on the jth motion state prediction information and sample labels corresponding to the multiple spatial grids in the third space at the first moment includes: determining the jth first sub-loss based on the jth motion state prediction information and motion state sample labels corresponding to the multiple spatial grids in the third space; determining the jth second sub-loss based on the jth occupancy state prediction information and occupancy state sample labels corresponding to the multiple spatial grids in the third space; determining the jth loss based on the jth first sub-loss and the jth second sub-loss. The first sub-loss and the second sub-loss may be cross entropy losses.

[0125] Exemplarily, based on the jth first sub-loss and the jth second sub-loss, the jth loss can be determined by taking the sum of the jth first sub-loss and the jth second sub-loss as the jth loss. It should be noted that the jth loss can also be calculated in other ways, such as calculating the average value, maximum value, etc. of the jth first sub-loss and the jth second sub-loss, which are not exhaustive here.

[0126] The training of the preset model based on the j-th loss to obtain the preset model after the j-th training may include: adjusting the parameters of the preset model based on the j-th loss reverse conduction; obtaining the preset model after the j-th training when the convergence condition is met. The convergence condition may be at least one of the following: the j-th loss is less than or equal to a threshold; the number of cyclic trainings reaches a preset number, etc. The threshold may be set according to actual conditions, and this application does not limit it.

[0127] The method of using the preset model after the j-th training as the target model when the j-th training is the last training among the multiple trainings includes: judging whether the j-th training is the last training among the multiple trainings, and using the preset model after the j-th training as the target model when the j-th training is the last training among the multiple trainings.

[0128] Whether the jth training is the last training can be determined according to a preset number of training times. For example, if the number of jth training times reaches the preset number of training times, the jth training can be determined to be the last training. Otherwise, it is determined that the jth training is not the last training. The preset number of training times can be configured according to actual conditions, for example, it can be 8 times, or 6 times, or larger or smaller, and this embodiment does not limit it.

[0129] In addition, the method may further include: when the j-th training is not the last training in the multiple trainings, performing the j+1-th training in the multiple trainings. The method of the j+1-th training in the multiple trainings is the same as the method of the j-th training in the multiple trainings, which will not be described in detail here.

[0130] It should be understood that the processing of obtaining the target model by training the preset model multiple times based on samples and sample labels provided in the aforementioned embodiment can be performed by an electronic device that executes the prediction method of the motion state corresponding to the spatial grid provided in this application. Alternatively, the preset model is trained multiple times based on samples and sample labels to obtain the target model, which may also be performed by other devices or equipment other than the electronic device. The method of obtaining the target model by performing training by other devices or equipment is the same as that in this embodiment, and will not be repeated. In the case where the target model is obtained by performing training by other devices or equipment, the electronic device can receive and save the target model from other devices or equipment in advance.

[0131] In this way, by training the preset model multiple times, and when the j-th training in the multiple trainings meets the convergence condition, and the j-th training is the last training in the multiple trainings, the preset model after the j-th training is used as the target model. In this way, the target model can predict the motion state information corresponding to multiple spatial grids in the space based on the image information, which can ensure the accuracy of the prediction of the motion state of the spatial grid by the trained target model. In addition, gradually adding noise coding to the second conditional vector (i.e., sample label) in different input times of the preset model can enable the preset model to learn the relationship between sample labels and samples of different clarity, and then when the model predicts, the input information of different clarity can be accurately predicted, thereby improving the accuracy of the model prediction.

[0132] Figure 3 FIG. 1 is a schematic block diagram of a device for predicting a motion state corresponding to a spatial grid provided by an embodiment of the present disclosure. Figure 3 As shown, including:

[0133] A feature extraction module 301 is used to determine feature vectors of multiple spatial grids in the first space based on multiple images of the first space collected at a current moment;

[0134] The prediction module 302 is used to perform multiple iterative processing on the feature vectors of multiple spatial grids in the first space and the motion state prediction information corresponding to the multiple spatial grids in the second space to obtain the motion state prediction information corresponding to the multiple spatial grids in the first space, wherein the motion state prediction information corresponding to the multiple spatial grids in the second space is obtained at a historical moment.

[0135] The prediction module is used to generate the information to be processed for the i-th iterative processing in the i-th iterative processing among the multiple iterative processings, based on the feature vectors of the multiple spatial grids in the first space and the motion state prediction information corresponding to the multiple spatial grids in the second space, wherein i is a positive integer; input the information to be processed for the i-th iterative processing and the i-th first conditional vector into the target model to obtain the i-th motion state prediction information corresponding to the multiple spatial grids in the first space output by the target model; and when the i-th iterative processing satisfies the constraint conditions, use the i-th motion state prediction information corresponding to the multiple spatial grids in the first space as the motion state prediction information corresponding to the multiple spatial grids in the first space.

[0136] Among them, when i is equal to 1, the i-th first conditional vector is a random noise vector; when i is greater than 1, the i-th first conditional vector is the i-1th motion state prediction information corresponding to multiple spatial grids in the first space.

[0137] like Figure 4 Said device further comprises:

[0138] The model training module 401 is used to perform multiple training on the preset model based on samples and sample labels to obtain the target model, wherein the samples include feature vectors of multiple spatial grids in the third space at a first moment and motion state information corresponding to multiple spatial grids in the fourth space at a second moment, and the first moment is later than the second moment.

[0139] The model training module is used to input the sample and the jth second conditional vector into the preset model in the jth training of the multiple trainings, and obtain the jth motion state prediction information corresponding to the multiple spatial grids in the third space at the first moment output by the preset model, wherein j is a positive integer; determine the jth loss based on the jth motion state prediction information and sample labels corresponding to the multiple spatial grids in the third space at the first moment; train the preset model based on the jth loss to obtain the preset model after the jth training; when the jth training is the last training among the multiple trainings, use the preset model after the jth training as the target model.

[0140] Among them, when j is equal to 1, the j-th second conditional vector is obtained by adding a random noise coding vector to the sample label; when j is greater than 1, the j-th second conditional vector is obtained by adding a random noise vector to the j-1-th second conditional vector.

[0141] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0142] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0143] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0144] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0145] like Figure 5As shown, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0146] Multiple components in the electronic device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the electronic device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0147] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above. For example, in some embodiments, the above method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, at least one step of the method described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform the above method in any other appropriate manner (e.g., by means of firmware).

[0148] According to another aspect of the present disclosure, an autonomous driving vehicle is provided, comprising the above-mentioned electronic device.

[0149] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0150] The program code for implementing the method of the present disclosure may be written in any combination of at least one programming language. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0151] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on at least one line, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0152] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0153] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0154] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0155] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0156] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for predicting a motion state corresponding to a spatial grid, comprising: Determine, based on a plurality of images of the first space collected at a current moment, feature vectors of a plurality of spatial grids in the first space; The feature vectors of the multiple spatial grids in the first space and the motion state prediction information corresponding to the multiple spatial grids in the second space are iterated multiple times to obtain the motion state prediction information corresponding to the multiple spatial grids in the first space, wherein the motion state prediction information corresponding to the multiple spatial grids in the second space is obtained at a historical moment.

2. The method according to claim 1, wherein: The step of performing multiple iterations on the feature vectors of the plurality of spatial grids in the first space and the motion state prediction information corresponding to the plurality of spatial grids in the second space to obtain the motion state prediction information corresponding to the plurality of spatial grids in the first space includes: In the i-th iterative processing of the multiple iterative processings, based on the feature vectors of the multiple spatial grids in the first space and the motion state prediction information corresponding to the multiple spatial grids in the second space, the to-be-processed information of the i-th iterative processing is generated, wherein i is a positive integer; Inputting the information to be processed in the i-th iteration and the i-th first condition vector into the target model, and obtaining the i-th motion state prediction information corresponding to the multiple spatial grids in the first space output by the target model; In a case where the i-th iterative processing satisfies the constraint condition, the i-th motion state prediction information corresponding to the multiple spatial grids in the first space is used as the motion state prediction information corresponding to the multiple spatial grids in the first space.

3. The method according to claim 2, wherein: When i is equal to 1, the i-th first condition vector is a random noise vector; When i is greater than 1, the i-th first conditional vector is the i-1th motion state prediction information corresponding to multiple space grids in the first space.

4. The method according to claim 2 or 3, further comprising: The preset model is trained multiple times based on samples and sample labels to obtain the target model, wherein the samples include feature vectors of multiple spatial grids in a third space at a first moment and motion state information corresponding to multiple spatial grids in a fourth space at a second moment, and the first moment is later than the second moment.

5. The method according to claim 4, wherein: The method of training the preset model multiple times based on samples and sample labels to obtain the target model includes: In the jth training of the multiple trainings, the sample and the jth second condition vector are input into a preset model to obtain the jth motion state prediction information corresponding to the plurality of spatial grids in the third space at the first moment output by the preset model, wherein j is a positive integer; Determine a j-th loss based on the j-th motion state prediction information and sample labels corresponding to the plurality of spatial grids in the third space at the first moment; Training the preset model based on the j-th loss to obtain the preset model after the j-th training; In the case that the j-th training is the last training among the multiple trainings, the preset model after the j-th training is used as the target model.

6. The method according to claim 5, wherein: When j is equal to 1, the j-th second conditional vector is obtained by adding a random noise encoding vector to the sample label; When j is greater than 1, the j-th second condition vector is obtained by adding a random noise vector to the j-1-th second condition vector.

7. A device for predicting a motion state corresponding to a spatial grid, comprising: A feature extraction module, used to determine feature vectors of a plurality of spatial grids in the first space based on a plurality of images of the first space collected at a current moment; A prediction module is used to perform multiple iterative processing on the feature vectors of multiple spatial grids in the first space and the motion state prediction information corresponding to multiple spatial grids in the second space to obtain the motion state prediction information corresponding to the multiple spatial grids in the first space, wherein the motion state prediction information corresponding to the multiple spatial grids in the second space is obtained at a historical moment.

8. The device according to claim 7, wherein: The prediction module is used to generate the information to be processed for the i-th iterative processing in the i-th iterative processing among the multiple iterative processings, based on the feature vectors of the multiple spatial grids in the first space and the motion state prediction information corresponding to the multiple spatial grids in the second space, wherein i is a positive integer; input the information to be processed for the i-th iterative processing and the i-th first conditional vector into the target model to obtain the i-th motion state prediction information corresponding to the multiple spatial grids in the first space output by the target model; and when the i-th iterative processing satisfies the constraint conditions, use the i-th motion state prediction information corresponding to the multiple spatial grids in the first space as the motion state prediction information corresponding to the multiple spatial grids in the first space.

9. The device according to claim 8, wherein: When i is equal to 1, the i-th first condition vector is a random noise vector; When i is greater than 1, the i-th first conditional vector is the i-1th motion state prediction information corresponding to multiple space grids in the first space.

10. The device according to claim 8 or 9, further comprising: A model training module is used to train a preset model multiple times based on samples and sample labels to obtain the target model, wherein the samples include feature vectors of multiple spatial grids in a third space at a first moment and motion state information corresponding to multiple spatial grids in a fourth space at a second moment, and the first moment is later than the second moment.

11. The device according to claim 10, wherein: The model training module is used to input the sample and the jth second conditional vector into the preset model in the jth training of the multiple trainings, and obtain the jth motion state prediction information corresponding to the multiple spatial grids in the third space at the first moment output by the preset model, wherein j is a positive integer; determine the jth loss based on the jth motion state prediction information and sample labels corresponding to the multiple spatial grids in the third space at the first moment; train the preset model based on the jth loss to obtain the preset model after the jth training; when the jth training is the last training among the multiple trainings, use the preset model after the jth training as the target model.

12. The device according to claim 11, wherein When j is equal to 1, the j-th second conditional vector is obtained by adding a random noise encoding vector to the sample label; When j is greater than 1, the j-th second condition vector is obtained by adding a random noise vector to the j-1-th second condition vector.

13. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-6.

15. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

16. An autonomous driving vehicle comprising the electronic device as claimed in claim 13.

Images