Method and apparatus for generating pseudo point cloud data, and terminal

Abstract

The application discloses a pseudo point cloud data generation method and device and a terminal. The method comprises the following steps: receiving a video stream acquired by a target vehicle at a target position; determining n depth maps corresponding to n continuous images and n-1 pose transformation estimations based on the video stream and a preset convolutional neural network model; determining initial pseudo point cloud data corresponding to the target vehicle based on the n depth maps and the n-1 pose transformation estimations; and merging and optimizing the initial pseudo point cloud data corresponding to multiple target vehicles to obtain target pseudo point cloud data corresponding to the target position. The application generates more accurate and complete target pseudo point cloud data by combining multiple initial pseudo point cloud data of multiple target vehicles and merging and optimizing the multiple initial pseudo point cloud data for the target position.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a pseudo point cloud data generation method and device and terminal. BACKGROUND

[0002] Scene three-dimensional reconstruction, that is, reconstructing three-dimensional space information of a real scene as a three-dimensional map of the scene. However, accurate pseudo point cloud data is an important basis for realizing three-dimensional reconstruction.

[0003] At present, large-scale scene reconstruction is supported Figure Three The generation method of pseudo point cloud data in three-dimensional reconstruction generally obtains point cloud data by scanning a scene offline by a laser radar (stationary or handheld), and then pre-processes and converts coordinates of the obtained point cloud data to obtain corresponding pseudo point cloud data.

[0004] However, the pseudo point cloud data generated by the above method has the problem of low accuracy. SUMMARY

[0005] The main purpose of the present application is to provide a pseudo point cloud data generation method and device and terminal to solve the problem of low accuracy of pseudo point cloud data in related technologies.

[0006] To achieve the above purpose, in a first aspect, the present application provides a pseudo point cloud data generation method, comprising:

[0007] receiving a video stream obtained by a target vehicle at a target position, wherein the video stream is n frames of continuous images, and n is an integer greater than 2;

[0008] determining n depth maps and n-1 pose transformation estimates corresponding to the n frames of continuous images based on the video stream and a preset convolutional neural network model, wherein the n frames of continuous images correspond to the n depth maps in one-to-one correspondence, and the pose transformation estimate is used to indicate a pose transformation matrix from a current frame image to a next frame image; Figure One

[0009] determining initial pseudo point cloud data corresponding to the target vehicle based on the n depth maps and the n-1 pose transformation estimates;

[0010] merging and optimizing the initial pseudo point cloud data corresponding to a plurality of target vehicles to obtain target pseudo point cloud data corresponding to the target position.

[0011] In a possible implementation manner, the determination of the n depth maps and the n-1 pose transformation estimates corresponding to the n frames of continuous images based on the video stream and the preset convolutional neural network model comprises:

[0012] ​input the video stream into a preset convolutional neural network model, and output n depth maps and n-1 pose transformation estimations corresponding to n continuous images when a target function of the preset convolutional neural network reaches a preset threshold, wherein the preset convolutional neural network model is obtained through training.

[0013] In a possible implementation, the initial pseudo point cloud data corresponding to the target vehicle is determined based on the n depth maps and the n-1 pose transformation estimations, including:

[0014] Each of the n depth maps is converted to obtain pseudo point cloud data corresponding to each of the n continuous images.

[0015] The initial pseudo point cloud data corresponding to the target vehicle is determined based on the pseudo point cloud data corresponding to each of the n continuous images and the n-1 pose transformation estimations.

[0016] In a possible implementation, the initial pseudo point cloud data corresponding to the target vehicle is determined based on the pseudo point cloud data corresponding to each of the n continuous images and the n-1 pose transformation estimations, including:

[0017] The pseudo point cloud data corresponding to each of the n continuous images is merged based on the n-1 pose transformation estimations to obtain pseudo point cloud data corresponding to the n continuous images.

[0018] The pseudo point cloud data corresponding to the n continuous images is optimized to obtain the initial pseudo point cloud data corresponding to the target vehicle.

[0019] In a possible implementation, the pseudo point cloud data corresponding to each of the n continuous images is merged based on the n-1 pose transformation estimations to obtain pseudo point cloud data corresponding to the n continuous images, including:

[0020] The pseudo point cloud data corresponding to each of the n continuous images is forwardly merged based on the n-1 pose transformation estimations to obtain pseudo point cloud data corresponding to the n continuous images.

[0021] In a possible implementation, the pseudo point cloud data corresponding to each of the n continuous images is forwardly merged based on the n-1 pose transformation estimations to obtain pseudo point cloud data corresponding to the n continuous images, including:

[0022] The pseudo point cloud data corresponding to the i th image of the n continuous images is multiplied by the last n-i pose transformation estimations of the n-1 pose transformation estimations to obtain pseudo point cloud data of the i th image in a coordinate system of the n th image, wherein i is an integer greater than or equal to 1 and less than n.

[0023] Merge the pseudo point cloud data of all the i-th image in the coordinate system of the n-th image and the pseudo point cloud data corresponding to the n-th image to obtain the pseudo point cloud data corresponding to the n continuous images.

[0024] In a possible implementation, the merging processing of the pseudo point cloud data corresponding to each image in the n continuous images according to the n-1 pose transformation estimations is performed to obtain the pseudo point cloud data corresponding to the n continuous images, including:

[0025] The inverse merging processing of the pseudo point cloud data corresponding to each image in the n continuous images according to the n-1 pose transformation estimations is performed to obtain the pseudo point cloud data corresponding to the n continuous images.

[0026] In a possible implementation, the merging processing of the pseudo point cloud data corresponding to each image in the n continuous images according to the n-1 pose transformation estimations is performed to obtain the pseudo point cloud data corresponding to the n continuous images, including:

[0027] Each of the n-1 pose transformation estimations is inversely transformed to obtain n-1 inverse transformed pose transformation estimations;

[0028] The pseudo point cloud data corresponding to the i-th image in the n continuous images is multiplied by the first i-1 pose transformation estimations in the n-1 pose transformation estimations to obtain the pseudo point cloud data of the i-th image in the coordinate system of the first image, where i is an integer greater than 1 and less than or equal to n;

[0029] Merge the pseudo point cloud data of all the i-th image in the coordinate system of the first image and the pseudo point cloud data corresponding to the first image to obtain the pseudo point cloud data corresponding to the n continuous images.

[0030] In a possible implementation, the initial pseudo point cloud data corresponding to the multiple target vehicles is merged and optimized to obtain the target pseudo point cloud data corresponding to the target position, including:

[0031] Obtain multiple initial positions corresponding to multiple target vehicles through a vehicle positioning device, where the multiple target vehicles and the multiple initial positions are in one-to-one correspondence.

[0032] When the multiple initial positions coincide, the initial pseudo point cloud data of the multiple initial positions is registered.

[0033] After the initial pseudo point cloud data of the multiple initial positions is successfully registered, the initial pseudo point cloud data is merged and optimized to obtain the initial pseudo point cloud data corresponding to the multiple target vehicles.

[0034] In the second aspect, an embodiment of the present application provides a pseudo point cloud data generation device, including:

[0035] receive a video stream acquired by a target vehicle at a target position, wherein the video stream is n frames of continuous images, and n is an integer greater than 2;

[0036] a model prediction module configured to determine n depth maps and n-1 pose transformation estimations corresponding to the n frames of continuous images based on the video stream and a preset convolutional neural network model, wherein the n frames of continuous images and the n depth maps correspond to each other, and the n-1 pose transformation estimations are used to indicate pose transformation matrices of a current frame image to a next frame image; Figure One a corresponding pair, the pose transformation estimation is used to indicate the pose transformation matrix of the current frame image to the next frame image;

[0037] an initial pseudo point cloud determination module configured to determine initial pseudo point cloud data corresponding to the target vehicle based on the n depth maps and the n-1 pose transformation estimations;

[0038] a target pseudo point cloud determination module configured to merge and optimize the initial pseudo point cloud data corresponding to the multiple target vehicles to obtain target pseudo point cloud data corresponding to the target position.

[0039] In a third aspect, an embodiment of the present application provides a terminal, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of any of the above pseudo point cloud data generation methods when executing the computer program.

[0040] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program implements the steps of any of the above pseudo point cloud data generation methods when executed by a processor.

[0041] The embodiment of the present application provides a pseudo point cloud data generation method, device and terminal, which comprises the following steps: receiving a video stream acquired by a target vehicle at a target position, wherein the video stream is n frames of continuous images; then determining n depth maps and n-1 pose transformation estimations corresponding to the n frames of continuous images based on the video stream and a preset convolutional neural network model; then determining initial pseudo point cloud data corresponding to the target vehicle based on the n depth maps and the n-1 pose transformation estimations; and finally merging and optimizing the initial pseudo point cloud data corresponding to the multiple target vehicles to obtain target pseudo point cloud data corresponding to the target position. The embodiment of the present application predicts the depth maps and the pose transformation estimations corresponding to the video stream based on the preset convolutional neural network model, and then generates the initial pseudo point cloud data of the target vehicle according to the depth maps and the pose transformation estimations, thereby improving the precision of the pseudo point cloud data. In addition, the embodiment of the present application combines the multiple initial pseudo point cloud data of the multiple target vehicles for the target position, and generates more accurate and complete target pseudo point cloud data by merging and optimizing the multiple initial pseudo point cloud data. BRIEF DESCRIPTION OF DRAWINGS

[0042] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application. The illustrations, together with the description, serve to explain the application, but are not intended to limit the application in any way. In the drawings:

[0043] Figure 1 is an implementation flowchart of a pseudo point cloud data generation method provided by an embodiment of the application;

[0044] Figure 2 is a structural schematic diagram of a pseudo point cloud data generation device provided by an embodiment of the application;

[0045] Figure 3 is a schematic diagram of a terminal provided by an embodiment of the application. DETAILED DESCRIPTION

[0046] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0047] The terms "first", "second", "third", "fourth" and the like (if any) in the description, claims and above drawings of the present application are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than that illustrated or described herein.

[0048] It should be understood that in various embodiments of the present application, the size of the serial number of each process does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0049] It should be understood that in the present application, "include" and "have" and any variants thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily limit to those clearly listed steps or units, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0050] It should be understood that in the present application, "a plurality of" means two or more. "And / or" is only a description of the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist together, and B exists alone. The character " / " generally represents that the associated objects before and after it are in an "or" relationship. "Including A, B and C", "including A, B, C" means that A, B and C are all included, "including A, B or C" means that one of A, B and C is included, and "including A, B and / or C" means that any one or any two or three of A, B and C is included.

[0051] It should be understood that in the present application, "B corresponding to A", "B corresponding to A", "A corresponding to B" or "B corresponding to A" means that B is associated with A, and B can be determined according to A. Determining B according to A does not mean that B is determined only according to A, but B can also be determined according to A and / or other information. The matching of A and B means that the similarity of A and B is greater than or equal to a preset threshold.

[0052] Depending on the context, "if" as used herein can be interpreted as "when" or "upon determining" or "in response to detecting".

[0053] The technical solutions of the present application will be described in detail below with specific examples. The following specific examples can be combined with each other, and the same or similar concepts or processes can not be described in some examples.

[0054] In order to make the purpose, technical solutions and advantages of the present application more clear, the following will be described by specific examples in combination with the drawings.

[0055] In one embodiment, as shown in Figure 1 A method for generating pseudo point cloud data is provided, comprising the following steps:

[0056] Step S101: receiving a video stream acquired by a target vehicle at a target position.

[0057] The video stream is n consecutive images, and n is an integer greater than 2.

[0058] The target vehicle in the present application is a vehicle equipped with an intelligent driving system, i.e. a positioning module and a vision module are arranged on the vehicle, wherein the positioning module includes but is not limited to GPS, Beidou, GLONASS, Galileo and quasi zenith, and the vision module includes an image acquisition unit and an image processing unit, and the image acquisition unit includes but is not limited to a vehicle-mounted camera.

[0059] Exemplarily, when the vehicle is driving at a target position, a video stream of the target position is acquired by a vehicle-mounted camera, and then the video stream is uploaded to the cloud, so that the cloud determines the pseudo point cloud data corresponding to the vehicle based on the video stream. Wherein, the target position refers to a certain region, such as Beijing or Changping District of Beijing, which is set according to specific circumstances.

[0060] Step S102: determining n depth maps and n-1 pose transformation estimates corresponding to n continuous images based on the video stream and the preset convolutional neural network model.

[0061] Wherein, the n continuous images correspond to the n depth Figure One maps and the n-1 pose transformation estimates, and the pose transformation estimate is used to indicate the pose transformation matrix from the current frame image to the next frame image.

[0062] When the cloud obtains the video stream corresponding to the target vehicle, the video stream needs to be input into the preset convolutional neural network model, and when the objective function of the preset convolutional neural network reaches the preset threshold, n depth maps and n-1 pose transformation estimates corresponding to n continuous images are output.

[0063] Wherein, the preset convolutional neural network model is obtained by training, that is, the preset convolutional neural network model is obtained after the convolutional neural network is trained by historical vehicle-mounted data. Wherein, the convolutional neural network (CNN) is a kind of feedforward neural network containing convolution calculation and having deep structure, which is one of the representative algorithms of deep learning. The convolutional neural network has representation learning ability and can perform translation invariant classification on input information according to its hierarchical structure, so that the preset convolutional neural network in the application has high prediction accuracy, can output accurate depth maps and pose transformation estimates, and further improve the generation accuracy of the pseudo point cloud data of the target vehicle.

[0064] Exemplarily, in the case of n = 3, when the cloud obtains the video stream corresponding to the target vehicle, the video stream includes 3 continuous frames of images, and then the 3 continuous frames of images are input into the preset convolutional neural network model, and when the target function of the preset convolutional neural network reaches a preset threshold, 3 depth maps corresponding to the 3 continuous frames of images and 2 pose transformation estimates are output. The 3 depth maps are respectively a first depth map A1 corresponding to a first frame of image, a second depth map A2 corresponding to a second frame of image, and a third depth map A3 corresponding to a third frame of image. The first pose transformation estimate in the 2 pose transformation estimates is a pose transformation matrix B1 from the first frame of image to the second frame of image, and the second pose transformation estimate is a pose transformation matrix B2 from the second frame of image to the third frame of image.

[0065] Step S103: determining initial pseudo point cloud data corresponding to the target vehicle based on the n depth maps and the n-1 pose transformation estimates.

[0066] After the n depth maps and the n-1 pose transformation estimates are predicted by the preset convolutional neural network model, each of the n depth maps needs to be converted to obtain pseudo point cloud data corresponding to each frame of image in the n continuous frames of images, and then the initial pseudo point cloud data corresponding to the target vehicle is determined according to the pseudo point cloud data corresponding to each frame of image in the n continuous frames of images and the n-1 pose transformation estimates.

[0067] On the basis of the above embodiment, after the 3 depth maps and the 2 pose transformation estimates are obtained, the first depth map A1, the second depth map A2 and the third depth map A3 in the 3 depth maps are converted respectively to obtain pseudo point cloud data corresponding to each frame of image, such as first pseudo point cloud data C1 corresponding to the first frame of image, second pseudo point cloud data C2 corresponding to the second frame of image, and third pseudo point cloud data C3 corresponding to the third frame of image.

[0068] After the pseudo point cloud data of each frame of image is obtained, the initial pseudo point cloud data corresponding to the target vehicle is determined according to the pseudo point cloud data and the n-1 pose transformation estimates corresponding to the video stream, that is, the pseudo point cloud data corresponding to each frame of image in the n continuous frames of images is merged according to the n-1 pose transformation estimates, to obtain pseudo point cloud data corresponding to the n continuous frames of images, and then the pseudo point cloud data corresponding to the n continuous frames of images is optimized to obtain the initial pseudo point cloud data corresponding to the target vehicle.

[0069] Since the n-1 pose transformation estimates corresponding to the video stream are obtained by using the above method, and the pose transformation estimates can be used to convert the pseudo point cloud data of the current frame of image into pseudo point cloud data in the coordinate system of the next frame of image, so as to facilitate the merging of all the frames of image in the video stream to form the pseudo point cloud data corresponding to the n continuous frames of images.

[0070] In a specific embodiment, the pseudo point cloud data corresponding to each of the n frames of continuous images can be forward combined according to the n-1 pose transformation estimations, to obtain the pseudo point cloud data corresponding to the n frames of continuous images. That is, the pseudo point cloud data corresponding to the i-th frame of the n frames of continuous images is multiplied by the last n-i pose transformation estimations of the n-1 pose transformation estimations, to obtain the pseudo point cloud data of the i-th frame in the coordinate system of the n-th frame of images, where i is an integer greater than or equal to 1 and less than n, and then the pseudo point cloud data of all the i-th frames in the coordinate system of the n-th frame of images and the pseudo point cloud data corresponding to the n-th frame of images are combined to obtain the pseudo point cloud data corresponding to the n frames of continuous images.

[0071] Exemplarily, in the case of n=3, the first pose transformation estimation (pose transformation matrix B1), the second pose transformation estimation (pose transformation matrix B2), and the first pseudo point cloud data C1, the second pseudo point cloud data C2 and the third pseudo point cloud data C3 have been determined by the above embodiment. Specifically, in the case of i=1, the first pseudo point cloud data C1 corresponding to the first frame of images is multiplied by the pose transformation matrix B1 and the pose transformation matrix B2, to obtain the pseudo point cloud data C1B1B2 of the first frame in the coordinate system of the third frame of images; in the case of i=2, the second pseudo point cloud data C2 corresponding to the second frame of images is multiplied by the pose transformation matrix B2, to obtain the pseudo point cloud data C2B2 of the second frame in the coordinate system of the third frame of images. Then the pseudo point cloud data C1B1B2 and C2B2 of the first frame and the second frame in the coordinate system of the third frame of images are added to the third pseudo point cloud data C3 corresponding to the third frame of images, to obtain the pseudo point cloud data X=C1B1B2+C2B2+C3 corresponding to the three frames of continuous images.

[0072] In another specific embodiment, the pseudo point cloud data corresponding to each of the n frames of continuous images can be inversely combined according to the n-1 pose transformation estimations, to obtain the pseudo point cloud data corresponding to the n frames of continuous images. That is, each of the n-1 pose transformation estimations is inversely transformed to obtain the inversely transformed n-1 pose transformation estimations, and then the pseudo point cloud data corresponding to the i-th frame of the n frames of continuous images is multiplied by the first i-1 pose transformation estimations of the n-1 pose transformation estimations, to obtain the pseudo point cloud data of the i-th frame in the coordinate system of the first frame of images, where i is an integer greater than 1 and less than or equal to n, and then the pseudo point cloud data of all the i-th frames in the coordinate system of the first frame of images and the pseudo point cloud data corresponding to the first frame of images are combined to obtain the pseudo point cloud data corresponding to the n frames of continuous images.

[0073] Exemplarily, in the case of n = 3, the first pose transformation estimation (pose transformation matrix B1), the second pose transformation estimation (pose transformation matrix B2), and the first pseudo point cloud data C1, the second pseudo point cloud data C2 and the third pseudo point cloud data C3 have been determined through the above embodiments. Specifically, the pose transformation matrix B1 and the pose transformation matrix B2 are first inversely transformed to obtain the pose transformation matrix B1 -1 and the pose transformation matrix B2 -1 , respectively. Then, in the case of i = 3, the third pseudo point cloud data C3 corresponding to the third frame image is multiplied by the pose transformation matrix B1 -1 and the pose transformation matrix B2 -1 to obtain the pseudo point cloud data C3B1 -1 B2 -1 of the third frame image in the coordinate system of the first frame image; in the case of i = 2, the second pseudo point cloud data C2 corresponding to the second frame image is multiplied by the pose transformation matrix B1 -1 to obtain the pseudo point cloud data C2B1 -1 of the second frame image in the coordinate system of the first frame image. Then, the pseudo point cloud data C3B1 1 B2 -1 and C2B1 -1 of the second and third frame images in the coordinate system of the first frame image are added to the first pseudo point cloud data C1 corresponding to the first frame image, i.e., the pseudo point cloud data X = C3B1 -1 B2 -1 + C2B1 -1 + C1 is obtained, which corresponds to the three consecutive frame images.

[0074] Step S104: merging and optimizing the initial pseudo point cloud data corresponding to the plurality of target vehicles to obtain the target pseudo point cloud data corresponding to the target position.

[0075] In order to obtain more accurate and complete target pseudo point cloud data corresponding to the target position, the present application adopts the case where a plurality of target vehicles drive at the target position, and then obtains the video stream of the target position through the vehicle-mounted camera of each target vehicle. Then, the plurality of video streams obtained by the plurality of target vehicles are uploaded to the cloud, and the cloud processes the video stream of each vehicle through the method of the above embodiments to generate the initial pseudo point cloud data corresponding to each target vehicle. Then, the plurality of initial positions corresponding to the plurality of target vehicles are obtained through the vehicle-mounted positioning device, wherein the plurality of target vehicles and the plurality of initial positions correspond one-to-one. When the plurality of initial positions overlap, the initial pseudo point cloud data of the plurality of initial positions is registered. After the initial pseudo point cloud data of the plurality of initial positions is successfully registered, the initial pseudo point cloud data is merged and optimized to obtain the initial pseudo point cloud data corresponding to the plurality of target vehicles. The optimization can be sampling, compression and other processing of the merged initial pseudo point cloud data.

[0076] The embodiment of the present application provides a pseudo point cloud data generation method, which comprises the following steps: receiving a video stream acquired by a target vehicle at a target position, wherein the video stream is n frames of continuous images; determining n depth maps corresponding to the n frames of continuous images and n-1 pose transformation estimations based on the video stream and a preset convolutional neural network model; determining initial pseudo point cloud data corresponding to the target vehicle based on the n depth maps and the n-1 pose transformation estimations; and merging and optimizing the initial pseudo point cloud data corresponding to a plurality of target vehicles to obtain target pseudo point cloud data corresponding to the target position. The preset convolutional neural network model is used to predict the depth maps and the pose transformation estimations corresponding to the video stream, and then the initial pseudo point cloud data of the target vehicle is generated according to the depth maps and the pose transformation estimations, so that the accuracy of the pseudo point cloud data is improved. In addition, the initial pseudo point cloud data of a plurality of target vehicles is combined for the target position, and the initial pseudo point cloud data is merged and optimized to generate more accurate and complete target pseudo point cloud data.

[0077] It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0078] The following is a device embodiment of the present application. For details not described in detail, reference can be made to the corresponding method embodiments described above.

[0079] Figure 2 A structure schematic diagram of a pseudo point cloud data generation device provided by the embodiment of the present application is shown. For the convenience of description, only the part related to the embodiment of the present application is shown. The pseudo point cloud data generation device comprises a receiving module 21, a model prediction module 22, an initial pseudo point cloud determination module 23 and a target pseudo point cloud determination module 24, and specifically as follows:

[0080] The receiving module 21 is used for receiving a video stream acquired by a target vehicle at a target position, wherein the video stream is n frames of continuous images, and n is an integer greater than 2;

[0081] The model prediction module 22 is used for determining n depth maps corresponding to the n frames of continuous images and n-1 pose transformation estimations based on the video stream and a preset convolutional neural network model, wherein the n frames of continuous images correspond to the n depth maps and the n-1 pose transformation estimations. Figure One The pose transformation estimation is used for indicating a pose transformation matrix from a current frame image to a next frame image;

[0082] The initial pseudo point cloud determination module 23 is used for determining initial pseudo point cloud data corresponding to the target vehicle based on the n depth maps and the n-1 pose transformation estimations;

[0083] The target pseudo point cloud determination module 24 is configured to merge and optimize the initial pseudo point cloud data corresponding to the target vehicle to obtain target pseudo point cloud data corresponding to the target position.

[0084] In a possible implementation, the model prediction module 22 comprises:

[0085] The model prediction sub-module is configured to input the video stream into a preset convolutional neural network model, and output n depth maps and n-1 pose transformation estimations corresponding to n continuous images when a target function of the preset convolutional neural network reaches a preset threshold, wherein the preset convolutional neural network model is obtained by training.

[0086] In a possible implementation, the initial pseudo point cloud determination module 23 comprises:

[0087] The conversion sub-module is configured to convert each of the n depth maps to obtain pseudo point cloud data corresponding to each of the n continuous images.

[0088] The initial pseudo point cloud determination sub-module is configured to determine the initial pseudo point cloud data corresponding to the target vehicle according to the pseudo point cloud data corresponding to each of the n continuous images and the n-1 pose transformation estimations.

[0089] In a possible implementation, the initial pseudo point cloud determination sub-module comprises:

[0090] The point cloud merging unit is configured to perform merging processing on the pseudo point cloud data corresponding to each of the n continuous images according to the n-1 pose transformation estimations to obtain pseudo point cloud data corresponding to the n continuous images.

[0091] The optimization unit is configured to perform optimization processing on the pseudo point cloud data corresponding to the n continuous images to obtain the initial pseudo point cloud data corresponding to the target vehicle.

[0092] In a possible implementation, the point cloud merging unit comprises:

[0093] The forward merging sub-unit is configured to perform forward merging processing on the pseudo point cloud data corresponding to each of the n continuous images according to the n-1 pose transformation estimations to obtain the pseudo point cloud data corresponding to the n continuous images.

[0094] In a possible implementation, the forward merging subunit is configured to multiply the pseudo point cloud data corresponding to the i-th image in the n continuous images by the last n-i pose transformation estimations in the n-1 pose transformation estimations, to obtain the pseudo point cloud data of the i-th image in the coordinate system of the n-th image, where i is an integer greater than or equal to 1 and less than n; and merge the pseudo point cloud data of all the i-th images in the coordinate system of the n-th image and the pseudo point cloud data corresponding to the n-th image, to obtain the pseudo point cloud data corresponding to the n continuous images.

[0095] In a possible implementation, the point cloud merging unit comprises:

[0096] The reverse merging subunit is configured to perform reverse merging processing on the pseudo point cloud data corresponding to each image in the n continuous images according to the n-1 pose transformation estimations, to obtain the pseudo point cloud data corresponding to the n continuous images.

[0097] In a possible implementation, the reverse merging subunit is configured to perform inverse transformation on each pose transformation estimation in the n-1 pose transformation estimations, to obtain the n-1 pose transformation estimations after inverse transformation; multiply the pseudo point cloud data corresponding to the i-th image in the n continuous images by the first i-1 pose transformation estimations in the n-1 pose transformation estimations, to obtain the pseudo point cloud data of the i-th image in the coordinate system of the first image, where i is an integer greater than 1 and less than or equal to n; and merge the pseudo point cloud data of all the i-th images in the coordinate system of the first image and the pseudo point cloud data corresponding to the first image, to obtain the pseudo point cloud data corresponding to the n continuous images.

[0098] In a possible implementation, the target pseudo point cloud determination module 24 comprises:

[0099] The initial position determination sub-module is configured to acquire a plurality of initial positions corresponding to a plurality of target vehicles by using a vehicle positioning device, where the plurality of target vehicles and the plurality of initial positions are in one-to-one correspondence.

[0100] The point cloud registration sub-module is configured to, when the plurality of initial positions coincide, register the initial pseudo point cloud data of the plurality of initial positions.

[0101] The target pseudo point cloud determination sub-module is configured to, after the registration of the initial pseudo point cloud data of the plurality of initial positions is successful, merge and optimize the initial pseudo point cloud data, to obtain the initial pseudo point cloud data corresponding to the plurality of target vehicles.

[0102] Figure 3 is a schematic diagram of a terminal provided by an embodiment of the present application. As shown in Figure 3As shown, the terminal 3 of this embodiment includes a processor 31, a memory 32, and a computer program 33 stored in the memory 32 and executable on the processor 31. The processor 31 implements the steps in the above various embodiments of the method for generating pseudo point cloud data when executing the computer program 33, for example Figure 1 The processor 31 implements the functions of the modules / units 21 to 24 in the above various embodiments of the apparatus for generating pseudo point cloud data when executing the computer program 33, for example Figure 2 The processor 31 implements the functions of the modules / units 21 to 24 in the above various embodiments of the apparatus for generating pseudo point cloud data when executing the computer program 33, for example

[0103] The present application also provides a readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the method for generating pseudo point cloud data provided by the various embodiments described above.

[0104] The readable storage medium can be a computer storage medium or a communication medium. The communication medium includes any medium that facilitates the transfer of a computer program from one place to another. The computer storage medium can be any available medium that can be accessed by a general or special purpose computer. For example, the readable storage medium is coupled to the processor, so that the processor can read information from and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). In addition, the ASIC can be located in a user equipment. Of course, the processor and the readable storage medium can also exist as discrete components in a communication device. The readable storage medium can be a read only memory (ROM), a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0105] The present application also provides a program product, which includes execution instructions stored in a readable storage medium. At least one processor of a device can read the execution instructions from the readable storage medium, and the at least one processor executes the execution instructions to make the device implement the method for generating pseudo point cloud data provided by the various embodiments described above.

[0106] In the embodiments of the above apparatus, it should be understood that the processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in combination with the present application can be directly embodied as hardware processor execution, or executed by a combination of hardware and software modules in the processor.

[0107] The above embodiments are only used to illustrate the technical solutions of the present application, rather than limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for generating pseudo point cloud data, characterized by, The method comprises the steps of: receiving a video stream obtained by a target vehicle at a target position, wherein the video stream is n frames of continuous images, and n is an integer greater than 2; based on the video stream and a preset convolutional neural network model, determining n depth maps corresponding to the n frames of continuous images and n-1 pose transformation estimates, wherein the n frames of continuous images correspond to the n depth maps one by one, and the pose transformation estimates are used to indicate the pose transformation matrix from the current frame image to the next frame image; based on the n depth maps and the n-1 pose transformation estimates, determining initial pseudo point cloud data corresponding to the target vehicle; merging and optimizing a plurality of initial pseudo point cloud data corresponding to the target vehicle to obtain target pseudo point cloud data corresponding to the target position; the step of determining the initial pseudo point cloud data corresponding to the target vehicle based on the n depth maps and the n-1 pose transformation estimates comprises the steps of: converting each depth map in the n depth maps to obtain pseudo point cloud data corresponding to each frame of image in the n frames of continuous images; determining the initial pseudo point cloud data corresponding to the target vehicle according to the pseudo point cloud data corresponding to each frame of image in the n frames of continuous images and the n-1 pose transformation estimates; the step of determining the initial pseudo point cloud data corresponding to the target vehicle according to the pseudo point cloud data corresponding to each frame of image in the n frames of continuous images and the n-1 pose transformation estimates comprises the steps of: performing forward merging processing on the pseudo point cloud data corresponding to each frame of image in the n frames of continuous images according to the n-1 pose transformation estimates to obtain pseudo point cloud data corresponding to the n frames of continuous images; the step of performing forward merging processing on the pseudo point cloud data corresponding to each frame of image in the n frames of continuous images according to the n-1 pose transformation estimates to obtain the pseudo point cloud data corresponding to the n frames of continuous images comprises the steps of: multiplying the pseudo point cloud data corresponding to the i-th frame of image in the n frames of continuous images and the last n-i pose transformation estimates in the n-1 pose transformation estimates to obtain the pseudo point cloud data of the i-th frame of image in the coordinate system of the n-th frame of image, wherein i is an integer greater than or equal to 1 and less than n; merging all the pseudo point cloud data of the i-th frame of image in the coordinate system of the n-th frame of image and the pseudo point cloud data corresponding to the n-th frame of image to obtain the pseudo point cloud data corresponding to the n frames of continuous images.

2. The method of claim 1, wherein the pseudo point cloud data is generated by: the step of determining the n depth maps corresponding to the n frames of continuous images and the n-1 pose transformation estimates based on the video stream and the preset convolutional neural network model comprises the steps of: inputting the video stream into the preset convolutional neural network model, and outputting the n depth maps corresponding to the n frames of continuous images and the n-1 pose transformation estimates when the objective function of the preset convolutional neural network reaches a preset threshold, wherein the preset convolutional neural network model is obtained by training.

3. The method of claim 1, wherein the pseudo point cloud data is generated by: the step of determining the initial pseudo point cloud data corresponding to the target vehicle according to the pseudo point cloud data corresponding to each frame of image in the n frames of continuous images and the n-1 pose transformation estimates comprises the steps of: merge pseudo point cloud data corresponding to each of the n frames of continuous images according to the n-1 pose transformation estimations, to obtain pseudo point cloud data corresponding to the n frames of continuous images; perform optimization processing on the pseudo point cloud data corresponding to the n frames of continuous images, to obtain initial pseudo point cloud data corresponding to the target vehicle.

4. The method of claim 3, wherein the pseudo point cloud data is generated by: The merging of the pseudo point cloud data corresponding to each of the n frames of continuous images according to the n-1 pose transformation estimations comprises: The merging of the pseudo point cloud data corresponding to each of the n frames of continuous images according to the n-1 pose transformation estimations comprises:

5. The method of claim 4, wherein the pseudo point cloud data is generated by: The merging of the pseudo point cloud data corresponding to each of the n frames of continuous images according to the n-1 pose transformation estimations comprises: perform inverse transformation on each of the n-1 pose transformation estimations, to obtain n-1 inverse pose transformation estimations; multiply the pseudo point cloud data corresponding to the i-th frame of the n frames of continuous images with the first i-1 pose transformation estimations in the n-1 pose transformation estimations, to obtain pseudo point cloud data of the i-th frame of continuous images in the coordinate system of the first frame of continuous images, wherein i is an integer greater than 1 and less than or equal to n; merge all the pseudo point cloud data of the i-th frame of continuous images in the coordinate system of the first frame of continuous images and the pseudo point cloud data corresponding to the first frame of continuous images, to obtain the pseudo point cloud data corresponding to the n frames of continuous images.

6. The method of claim 1, wherein the pseudo point cloud data is generated by: The merging and optimization of the initial pseudo point cloud data corresponding to the multiple target vehicles to obtain the target pseudo point cloud data corresponding to the target position comprises: obtain multiple initial positions corresponding to multiple target vehicles through a vehicle-mounted positioning device, wherein the multiple target vehicles and the multiple initial positions correspond to each other in a one-to-one manner; when there is overlap among the multiple initial positions, register initial pseudo point cloud data of the multiple initial positions; after the registration of the initial pseudo point cloud data of the multiple initial positions is successful, merge and optimize the initial pseudo point cloud data, to obtain initial pseudo point cloud data corresponding to the multiple target vehicles.

7. A terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor implements the steps of the pseudo point cloud data generation method according to any one of claims 1 to 6 when executing the computer program.

Images