Data mapping method and apparatus, computer device, medium, and program product

Abstract

The application relates to a data mapping method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: obtaining a two-dimensional coordinate of a ground positioning point of a to-be-detected target in a scene image of a target scene according to the to-be-detected target in the scene image, and then mapping the two-dimensional coordinate of the ground positioning point of the to-be-detected target into a three-dimensional coordinate according to a preset coordinate mapping relationship. The coordinate mapping relationship is established based on two-dimensional pixel points and an expanded ground point cloud of the target scene. The expanded ground point cloud can cover the ground area in the entire target scene to the maximum extent, so as to establish the corresponding relationship between the two-dimensional pixel points and the ground point cloud of the target scene, that is, the above-mentioned preset coordinate mapping relationship, improve the comprehensiveness of data mapping, and then improve the accuracy of data mapping, and accordingly improve the accuracy of target information detection based on multi-sensor data fusion.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data mapping method, apparatus, computer equipment, medium, and program product. Background Technology

[0002] Multi-sensor fusion is a commonly used technique in environmental perception and target localization.

[0003] For example, vehicles can detect target information based on the fusion of camera and radar. In the process of camera and radar fusion, 3D and 2D spatial coordinate transformation is involved. That is, the position information of the target in three-dimensional space can be determined based on the image acquired by the camera, so as to realize the target information detection.

[0004] However, the accuracy of target location information in three-dimensional space obtained from images captured by cameras in related technologies is poor, resulting in poor accuracy of target information detection. Summary of the Invention

[0005] Therefore, it is necessary to provide a data mapping method, apparatus, computer equipment, computer-readable storage medium, and computer program product to address the aforementioned technical problems.

[0006] Firstly, this application provides a data mapping method, including:

[0007] Based on the target to be detected in the scene image of the target scene, obtain the two-dimensional coordinates of the ground positioning point of the target to be detected in the scene image;

[0008] According to the preset coordinate mapping relationship, the two-dimensional coordinates of the ground positioning points of the target to be detected are mapped to three-dimensional coordinates; wherein, the coordinate mapping relationship is established based on the two-dimensional pixels of the target scene and the expanded ground point cloud.

[0009] In one embodiment, based on the target to be detected in the scene image of the target scene, the two-dimensional coordinates of the ground positioning point of the target to be detected in the scene image are obtained, including:

[0010] Obtain the annotation data of the target to be detected in the scene image;

[0011] Based on the labeled data, determine the two-dimensional coordinates of the ground positioning point of the target to be detected in the scene image.

[0012] In one embodiment, the annotation data includes the coordinates of the center point of the annotation box and the size of the annotation box; based on the annotation data, determining the two-dimensional coordinates of the ground positioning point of the target to be detected in the scene image includes:

[0013] Input the coordinates of the center point of the annotation box and the size of the annotation box into the reference neural network model to obtain the reference ratio output by the reference neural network model; the reference ratio characterizes the degree of positional deviation between the center point of the annotation box and the ground positioning point of the target to be detected;

[0014] Based on the coordinates of the center point of the annotation box, the size of the annotation box, and the reference ratio, determine the two-dimensional coordinates of the ground positioning point of the target to be detected in the scene image.

[0015] In one embodiment, the reference ratio includes a horizontal coordinate ratio and a vertical coordinate ratio; then, based on the coordinates of the center point of the bounding box, the size of the bounding box, and the reference ratio, the two-dimensional coordinates of the ground positioning point of the target to be detected in the scene image are determined, including:

[0016] The x-coordinate of the ground positioning point in the scene image is obtained by using the x-coordinate of the center point coordinate of the annotation box, the length of the annotation box in the x-coordinate direction in the annotation box size, and the x-coordinate ratio.

[0017] The ordinate of the ground positioning point in the scene image is obtained by using the ordinate of the center point coordinate of the annotation box, the length of the annotation box in the ordinate direction in the annotation box size, and the ratio of the ordinate.

[0018] In one embodiment, the method further includes:

[0019] Obtain the original point cloud set and original scene image of the target scene;

[0020] Extract the ground point cloud from the original point cloud set and expand the ground point cloud to obtain the expanded ground point cloud;

[0021] A coordinate mapping relationship is generated based on the expanded ground point cloud and the original scene image.

[0022] In one embodiment, generating a coordinate mapping relationship based on the augmented ground point cloud and the original scene image includes:

[0023] Based on the calibration relationship between the lidar and the camera equipment, the augmented ground points are converted into the original scene image, resulting in the converted pixels of the augmented ground point cloud in the original scene image.

[0024] Match the transformed pixels with the two-dimensional pixels in the original scene image;

[0025] The successfully matched 2D pixels are associated one-to-one with the corresponding expanded ground point cloud to obtain the coordinate mapping relationship.

[0026] In one embodiment, the method further includes:

[0027] Obtain the labeled data of multiple sample images; input the labeled data of each sample image into the initial reference neural network model to obtain the initial reference ratio output by the initial reference neural network model;

[0028] Compare the corresponding initial reference ratio with the target reference ratio to obtain the comparison result;

[0029] Adjust the model parameters in the initial reference neural network model based on the comparison results of each sample image until the model converges, thus obtaining the reference neural network model.

[0030] Secondly, this application also provides a data mapping apparatus, comprising:

[0031] The two-dimensional coordinate determination module is used to obtain the two-dimensional coordinates of the ground positioning point of the target to be detected in the scene image based on the target to be detected in the scene image of the target scene;

[0032] The three-dimensional coordinate determination module is used to map the two-dimensional coordinates of the ground positioning points of the target to be detected into three-dimensional coordinates according to the preset coordinate mapping relationship; wherein, the coordinate mapping relationship is established based on the two-dimensional pixels of the target scene and the expanded ground point cloud.

[0033] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0034] Based on the target to be detected in the scene image of the target scene, obtain the two-dimensional coordinates of the ground positioning point of the target to be detected in the scene image;

[0035] According to the preset coordinate mapping relationship, the two-dimensional coordinates of the ground positioning points of the target to be detected are mapped to three-dimensional coordinates; wherein, the coordinate mapping relationship is established based on the two-dimensional pixels of the target scene and the expanded ground point cloud.

[0036] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0037] Based on the target to be detected in the scene image of the target scene, obtain the two-dimensional coordinates of the ground positioning point of the target to be detected in the scene image;

[0038] According to the preset coordinate mapping relationship, the two-dimensional coordinates of the ground positioning points of the target to be detected are mapped to three-dimensional coordinates; wherein, the coordinate mapping relationship is established based on the two-dimensional pixels of the target scene and the expanded ground point cloud.

[0039] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0040] Based on the target to be detected in the scene image of the target scene, obtain the two-dimensional coordinates of the ground positioning point of the target to be detected in the scene image;

[0041] According to the preset coordinate mapping relationship, the two-dimensional coordinates of the ground positioning points of the target to be detected are mapped to three-dimensional coordinates; wherein, the coordinate mapping relationship is established based on the two-dimensional pixels of the target scene and the expanded ground point cloud.

[0042] The aforementioned data mapping method, apparatus, computer equipment, storage medium, and computer program product acquire the two-dimensional coordinates of the ground positioning points of the target to be detected in the scene image of the target scene, and then map the two-dimensional coordinates of the ground positioning points of the target to be detected into three-dimensional coordinates according to a preset coordinate mapping relationship. The coordinate mapping relationship is established based on the two-dimensional pixels of the target scene and the expanded ground point cloud. The expanded ground point cloud can cover the ground area of ​​the entire target scene to the greatest extent, establishing a more comprehensive correspondence between the two-dimensional pixels of the target scene and the ground point cloud, i.e., the aforementioned preset coordinate mapping relationship. This ensures that all two-dimensional pixels belonging to the ground area in the scene image have a corresponding ground point cloud and obtain corresponding three-dimensional coordinates, improving the comprehensiveness of the data mapping. It eliminates the need to use the three-dimensional coordinates of the ground point cloud corresponding to other two-dimensional pixels to estimate the three-dimensional coordinates of two-dimensional pixels without a corresponding ground point cloud, thereby improving the accuracy of the data mapping and consequently improving the accuracy of subsequent target information detection based on multi-sensor data fusion. Attached Figure Description

[0043] Figure 1 This is a flowchart illustrating a data mapping method in one embodiment;

[0044] Figure 2 This is a flowchart illustrating the process of determining the two-dimensional coordinates of a ground positioning point in a scene image based on annotation information in one embodiment.

[0045] Figure 3 This is a flowchart illustrating the process of determining the two-dimensional coordinates of a ground positioning point in a scene image based on a parametric neural network model in one embodiment.

[0046] Figure 4 This is a flowchart illustrating the process of determining the two-dimensional coordinates of a ground positioning point in a scene image, as described in another embodiment.

[0047] Figure 5 This is a schematic diagram of the process for generating coordinate mapping relationships in one embodiment;

[0048] Figure 6 This is a flowchart illustrating the process of generating coordinate mapping relationships in another embodiment;

[0049] Figure 7 This is a flowchart illustrating the process of obtaining a reference neural network model in one embodiment;

[0050] Figure 8 This is a structural block diagram of a data mapping device in one embodiment;

[0051] Figure 9 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0053] In one embodiment, such as Figure 1 As shown, a data mapping method is provided. This embodiment illustrates the application of this method to a terminal. It can be understood that the terminal can be a roadside device, vehicle, drone, etc., that is, a terminal device used to achieve data fusion between multiple sensors. In this embodiment, the method includes the following steps:

[0054] S110. Based on the target to be detected in the scene image of the target scene, obtain the two-dimensional coordinates of the ground positioning point of the target to be detected in the scene image.

[0055] In this context, the scene image of the target scene is a two-dimensional image of the target scene; the target to be detected is the target object in the target scene, which can be any object appearing in the target scene, such as a person, vehicle, or building. The ground positioning point of the target to be detected is the pixel in the scene image used to represent the position of the target to be detected.

[0056] Optionally, the ground location point of the target to be detected can be any pixel point in the scene image where the target to be detected contacts the ground area, such as the contact center point.

[0057] Optionally, the terminal acquires a scene image of the target scene through a camera device and detects the target to be detected in the scene image, so as to further obtain the two-dimensional coordinates of the ground positioning point of the target to be detected in the scene image.

[0058] S120. Based on the preset coordinate mapping relationship, the two-dimensional coordinates of the ground positioning point of the target to be detected are mapped to three-dimensional coordinates.

[0059] The coordinate mapping relationship is established based on the two-dimensional pixels of the target scene and the expanded ground point cloud. The ground point cloud refers to the point cloud data of the target scene acquired by the LiDAR that belongs to the ground region. The coordinate mapping relationship is used to characterize the correspondence between the two-dimensional pixels in the scene image of the target scene and the ground point cloud in the three-dimensional space of the target scene.

[0060] Due to the sparsity of point clouds and the occlusion of ground areas by target objects or other background environments in the target scene, the actual ground point cloud collected is relatively small. After expanding the data volume of the actual collected ground point cloud, the expanded ground point cloud is obtained.

[0061] Specifically, after obtaining the two-dimensional coordinates of the ground positioning point of the target to be detected in the scene image, the terminal can determine the ground point cloud corresponding to the two-dimensional pixel according to the preset coordinate mapping relationship, so as to obtain the three-dimensional coordinates of the ground point cloud, that is, to realize the mapping of the two-dimensional coordinates of the ground positioning point of the target to be detected to three-dimensional coordinates.

[0062] In this embodiment, the terminal obtains the two-dimensional coordinates of the ground positioning points of the target to be detected in the scene image of the target scene. Then, according to a preset coordinate mapping relationship, the two-dimensional coordinates of the ground positioning points of the target to be detected are mapped to three-dimensional coordinates. The coordinate mapping relationship is established based on the two-dimensional pixels of the target scene and the expanded ground point cloud. The expanded ground point cloud can cover the ground area of ​​the entire target scene to the greatest extent, establishing a more comprehensive correspondence between the two-dimensional pixels of the target scene and the ground point cloud, i.e., the aforementioned preset coordinate mapping relationship. This ensures that all two-dimensional pixels belonging to the ground area in the scene image have a corresponding ground point cloud and obtain corresponding three-dimensional coordinates, improving the comprehensiveness of data mapping. It eliminates the need to use the three-dimensional coordinates of the ground point cloud corresponding to other two-dimensional pixels to estimate the three-dimensional coordinates of two-dimensional pixels without a corresponding ground point cloud, thereby improving the accuracy of data mapping and consequently improving the accuracy of subsequent target information detection based on multi-sensor data fusion.

[0063] In one embodiment, such as Figure 2 As shown, S110 above, obtaining the two-dimensional coordinates of the ground positioning point of the target to be detected in the scene image of the target scene, includes:

[0064] S210. Obtain the annotation data of the target to be detected in the scene image.

[0065] The annotation data can be used to determine the location of the ground coordinates of the target object in the scene image. Optionally, the annotation data can be the two-dimensional coordinates of the ground coordinates, or the position and size information of the annotation box. The annotation box is the bounding box in the scene image that defines the area where the target object is located.

[0066] Optionally, after the terminal obtains the target to be detected in the detected scene image, it uses annotation information to annotate the target to be detected. For example, it directly annotates the two-dimensional coordinates of the ground positioning point of the target to be detected and uses the two-dimensional coordinates of the ground positioning point as the above annotation data, or it uses a bounding box to frame the area where the target to be detected is located and uses the position and size information of the bounding box as the above annotation data.

[0067] S220. Based on the labeled data, determine the two-dimensional coordinates of the ground positioning point of the target to be detected in the scene image.

[0068] Optionally, when the annotation data is the two-dimensional coordinates of the ground positioning point of the target to be detected in the scene image, the terminal directly obtains the two-dimensional coordinates of the ground positioning point of the target to be detected in the scene image after acquiring the annotation data. When the annotation data is the position and size information of the annotation box, the terminal calculates the two-dimensional coordinates of the ground positioning point of the target to be detected in the scene image based on the position and size information of the annotation box.

[0069] In this embodiment, the terminal acquires the annotation data of the target to be detected in the scene image, and determines the two-dimensional coordinates of the ground positioning point of the target to be detected in the scene image based on the annotation data, so as to accurately determine the position of the ground positioning point of the target to be detected in the scene image, which is beneficial to the accuracy of subsequent data mapping.

[0070] In practical applications, the 3D coordinates of the near-ground center point of the target object in 3D space are generally selected to represent the spatial position of the target object in the target scene. However, the near-ground center point of the target object in the scene image does not accurately correspond to the actual near-ground center point of the target object in 3D space. If the near-ground center point of the target object in the scene image is directly mapped to the near-ground center point of the target object in the target scene, there will be a large mapping error. Therefore, in order to determine the 3D coordinates of the near-ground center point of the target object in 3D space based on the above data mapping method, it is necessary to determine the optimal mapping point in the scene image to reduce mapping error and improve mapping accuracy. The following embodiment describes the process of determining the optimal mapping point in the scene image.

[0071] In one embodiment, such as Figure 3As shown, the annotation data includes the coordinates of the center point of the annotation box and the size of the annotation box. The above-mentioned step S220, determining the two-dimensional coordinates of the ground positioning point of the target to be detected in the scene image based on the annotation data, includes:

[0072] S310. Input the coordinates of the center point of the annotation box and the size of the annotation box into the reference neural network model to obtain the reference ratio output by the reference neural network model.

[0073] The coordinates of the center point of the bounding box represent the center position of the target object selected by the bounding box in the scene image. The reference ratio represents the degree of positional deviation between the center point of the bounding box and the ground positioning point of the target to be detected.

[0074] It should be noted that the aforementioned reference neural network model is determined based on bounding box data from a large number of sample images and the two-dimensional coordinates of the optimal mapping point in the corresponding sample images. The optimal mapping point is the ground positioning point corresponding to the true near-ground center point of the target object in the three-dimensional space of the target scene. Specifically, the reference neural network model is used to determine the aforementioned reference ratio based on the coordinates of the bounding box center point and the bounding box size. This reference ratio is determined based on the coordinates of the bounding box center point, the bounding box size, and the two-dimensional coordinates of the optimal mapping point in the corresponding image. Therefore, given the coordinates of the bounding box center point, the bounding box size, and this reference ratio of the target object in the scene image, the two-dimensional coordinates of the ground positioning point in the target scene, which serves as the optimal mapping point, can be calculated.

[0075] S320. Based on the coordinates of the center point of the annotation box, the size of the annotation box, and the reference ratio, determine the two-dimensional coordinates of the ground positioning point of the target to be detected in the scene image.

[0076] Specifically, based on the calculation method of the reference ratio, the terminal substitutes the coordinates of the center point of the above-mentioned annotation box, the size of the annotation box, and the reference ratio to obtain the two-dimensional coordinates of the ground positioning point (i.e. the best mapping point) of the target to be detected in the scene image.

[0077] In an alternative embodiment, such as Figure 4 As shown, the reference ratio includes the ratio of the horizontal coordinate and the ratio of the vertical coordinate. The above-mentioned S320, determining the two-dimensional coordinates of the ground positioning point of the target to be detected in the scene image based on the coordinates of the center point of the annotation box, the size of the annotation box, and the reference ratio, includes:

[0078] S410. Based on the x-coordinate of the center point coordinates of the annotation box, the length of the annotation box in the x-coordinate direction in the annotation box size, and the x-coordinate ratio, obtain the x-coordinate of the ground positioning point in the scene image.

[0079] The ratio of the horizontal axis, ratio_x, can be calculated using the following formula:

[0080]

[0081] W represents the length of the annotation box along the horizontal axis in the annotation box size, and d_x represents the horizontal coordinate deviation between the horizontal coordinate X' of the ground positioning point in the scene image and the horizontal coordinate X of the center point of the annotation box.

[0082] Specifically, the terminal inputs the x-coordinate X of the center point coordinates of the annotation box, the length W of the annotation box in the x-coordinate direction in the annotation box size, and the x-coordinate ratio ratio_x into the above calculation formula to obtain the x-coordinate X' of the ground positioning point in the scene image.

[0083] S420. Based on the ordinate of the center point coordinates of the annotation box, the length of the annotation box in the ordinate direction in the annotation box size, and the ratio of the ordinates, obtain the ordinate of the ground positioning point in the scene image.

[0084] The ratio of the horizontal axis, ratio_y, can be calculated using the following formula:

[0085]

[0086] Z represents the length of the annotation box along the vertical axis in the annotation box size, and d_y represents the vertical coordinate deviation between the vertical coordinate Y' of the ground positioning point in the scene image and the vertical coordinate Y of the center point of the annotation box.

[0087] Specifically, the terminal inputs the ordinate Y of the center point coordinates of the annotation box, the length Z of the annotation box in the ordinate direction of the annotation box size, and the ordinate ratio ratio_y into the above calculation formula to obtain the ordinate Y' of the ground positioning point in the scene image.

[0088] The x-coordinate X' and y-coordinate Y' of the ground positioning point in the scene image constitute the two-dimensional coordinates (X', Y') of the ground positioning point in the scene image.

[0089] In this embodiment, the terminal acquires the annotation data of the target to be detected in the scene image, and determines the two-dimensional coordinates of the ground positioning point of the target to be detected in the scene image based on the annotation data. Further, a reference neural network model is used to output a reference ratio based on the coordinates of the center point of the bounding box of the target to be detected in the scene image and the size of the bounding box. The two-dimensional coordinates of the ground positioning point of the target to be detected in the scene image are obtained according to the calculation method of the reference ratio. Since the above-mentioned reference neural network model is determined based on the bounding box data in a large number of sample images and the two-dimensional coordinates of the best mapping point in the corresponding sample images, the ground positioning point determined based on the reference ratio output by the reference neural network model is the aforementioned best mapping point. After the above data mapping, the actual near-ground center point of the target to be detected in three-dimensional space can be obtained from the best mapping point, thereby reducing mapping errors and improving the accuracy of data mapping.

[0090] In one embodiment, the method further includes a process for generating the aforementioned coordinate mapping relationship. For example... Figure 5 As shown, the above method also includes:

[0091] S510: Obtain the original point cloud set and original scene image of the target scene.

[0092] Optionally, the original scene image can be the same as the aforementioned scene image, or it can be an image acquired at a different time than the aforementioned scene image. For example, it could be an image acquired by the camera when the target object is not present in the target scene.

[0093] Specifically, the terminal acquires the original point cloud set of the target scene through LiDAR and acquires the original scene image of the target scene through camera equipment.

[0094] S520. Extract the ground point cloud from the original point cloud set and expand the ground point cloud to obtain the expanded ground point cloud.

[0095] Optionally, for the original point cloud set, the terminal uses a random sampling consistency sampling method to obtain the planar equation of the ground point cloud, and determines the ground point cloud in the original point cloud set based on a preset threshold. Furthermore, it expands the ground point cloud data based on the three-dimensional coordinates of the ground point cloud to obtain the expanded ground point cloud. Since the ground flatness of urban road scenes is relatively high, a piecewise trispline interpolation method or an adjacent interpolation method can be used to expand the ground point cloud.

[0096] S530: Generate coordinate mapping relationship based on the expanded ground point cloud and the original scene image.

[0097] Specifically, the terminal converts the expanded ground point cloud into the original scene image and maps it one by one with the two-dimensional pixels in the original scene image, thereby generating the coordinate mapping relationship between the two-dimensional pixels in the scene image of the target scene and the expanded ground point cloud in the three-dimensional space of the target scene.

[0098] In an alternative embodiment, such as Figure 6 As shown, the above-mentioned S540, generating a coordinate mapping relationship based on the expanded ground point cloud and the original scene image, includes:

[0099] S610. Based on the calibration relationship between the lidar and the camera equipment, the expanded ground points are converted into the original scene image to obtain the converted pixel points of the expanded ground point cloud in the original scene image.

[0100] Before determining the aforementioned coordinate mapping relationship, the laser type and camera equipment need to be synchronously calibrated to achieve time and spatial synchronization of data. The calibration relationship between the laser radar and camera equipment can be obtained based on the calibration of their intrinsic and extrinsic parameters.

[0101] Specifically, the terminal converts the expanded ground points into the original scene image based on the calibration relationship between the aforementioned lidar and camera equipment, so as to convert the three-dimensional coordinates of the expanded ground point cloud into two-dimensional coordinates, and obtain the converted pixel points of the expanded ground point cloud in the original scene image.

[0102] S620: Match the converted pixels with the two-dimensional pixels in the original scene image.

[0103] S630. Associate the successfully matched two-dimensional pixels with the corresponding expanded ground point cloud to obtain the coordinate mapping relationship.

[0104] Specifically, the terminal matches the two-dimensional coordinates of the converted pixels with the two-dimensional coordinates of the pixels in the original scene image. This allows for a one-to-one correspondence between the successfully matched two-dimensional pixels and the corresponding expanded ground point cloud. This results in the mapping relationship between the two-dimensional pixels in the scene image of the target scene and the expanded ground point cloud in the three-dimensional space of the target scene, i.e., the coordinate mapping relationship mentioned above.

[0105] In this embodiment, the terminal acquires the original point cloud set and the original scene image of the target scene, extracts the ground point cloud from the original point cloud set, and expands the ground point cloud to obtain the expanded ground point cloud. Then, a coordinate mapping relationship is generated based on the expanded ground point cloud and the original scene image. This method achieves a one-to-one mapping between the ground point cloud in the three-dimensional space of the target scene and the two-dimensional pixels in the scene image of the target scene, obtaining a coordinate mapping relationship that covers as much of the ground area as possible. This improves the comprehensiveness of the data mapping, thereby enhancing the accuracy of subsequent data mapping.

[0106] In one embodiment, the method further includes a process of training a reference neural network model.

[0107] like Figure 7 As shown, the above method also includes:

[0108] S710. Obtain the annotation data of multiple sample images.

[0109] As mentioned earlier, the annotation data includes the coordinates of the center point of the annotation box and the size of the annotation box.

[0110] Optionally, the above sample images can be divided into training samples and validation samples according to a certain ratio, and the training samples can be saved as train.pkl files, while the validation samples can be saved as val.pkl files.

[0111] S720. Input the labeled data of each sample image into the initial reference neural network model to obtain the initial reference ratio output by the initial reference neural network model.

[0112] Specifically, the terminal can read the training samples in the train.pkl file to input the coordinates of the center point of the bounding box and the size of the bounding box in the sample image used as the training samples into the initial reference neural network model to obtain the initial reference ratio output by the initial reference neural network model.

[0113] S730. Compare the corresponding initial reference ratio and the target reference ratio to obtain the comparison result.

[0114] S740. Adjust the model parameters in the initial reference neural network model according to the comparison results of each sample image until the model converges to obtain the reference neural network model.

[0115] The initial reference ratio is a parameter output by the initial reference neural network model, and the target reference ratio is the true value of the parameter calculated using the above-mentioned reference ratio calculation method based on the bounding box data in the corresponding sample image and the two-dimensional coordinates of the best mapping point.

[0116] Specifically, the terminal compares the corresponding initial reference ratio with the target reference ratio to obtain the reference ratio deviation, i.e., the comparison result mentioned above. Then, based on the comparison results of each sample image, the model parameters in the initial reference neural network model are adjusted until the model converges, thus obtaining the reference neural network model. Specifically, the model parameters in the initial reference neural network model are adjusted in the direction of reducing the aforementioned reference ratio deviation.

[0117] In addition to obtaining the aforementioned reference neural network model, the validation samples in the val.pkl file can be read to validate the reference neural network model, thereby ensuring the accuracy of the model.

[0118] In this embodiment, the terminal acquires labeled data from multiple sample images, inputs the labeled data of each sample image into an initial reference neural network model, obtains the initial reference ratio output by the initial reference neural network model, and then compares the corresponding initial reference ratio with the target reference ratio to obtain the comparison result. Based on the comparison result of each sample image, the model parameters in the initial reference neural network model are adjusted until the model converges, resulting in the reference neural network model. The two-dimensional coordinates of the optimal mapping point in the scene image can be determined based on the reference ratio output by the aforementioned reference neural network model, thereby improving the accuracy of subsequent data mapping.

[0119] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0120] In one embodiment, such as Figure 8 As shown, a data mapping device is provided, including: a two-dimensional coordinate determination module 801 and a three-dimensional coordinate determination module 802, wherein:

[0121] The two-dimensional coordinate determination module 801 is used to obtain the two-dimensional coordinates of the ground positioning point of the target to be detected in the scene image based on the target to be detected in the scene image of the target scene;

[0122] The three-dimensional coordinate determination module 802 is used to map the two-dimensional coordinates of the ground positioning points of the target to be detected into three-dimensional coordinates according to the preset coordinate mapping relationship; wherein, the coordinate mapping relationship is established based on the two-dimensional pixels of the target scene and the expanded ground point cloud.

[0123] In one embodiment, the two-dimensional coordinate determination module 801 is specifically used for:

[0124] Obtain the annotation data of the target to be detected in the scene image; based on the annotation data, determine the two-dimensional coordinates of the ground positioning point of the target to be detected in the scene image.

[0125] In one embodiment, the annotation data includes the coordinates of the center point of the annotation box and the size of the annotation box; the two-dimensional coordinate determination module 801 is specifically used for:

[0126] The coordinates of the center point of the annotation box and the size of the annotation box are input into the reference neural network model to obtain the reference ratio output by the reference neural network model. The reference ratio represents the degree of positional deviation between the center point of the near-ground border of the annotation box and the ground positioning point of the target to be detected. Based on the coordinates of the center point of the annotation box, the size of the annotation box and the reference ratio, the two-dimensional coordinates of the ground positioning point of the target to be detected in the scene image are determined.

[0127] In one embodiment, the reference ratio includes a horizontal coordinate ratio and a vertical coordinate ratio; the two-dimensional coordinate determination module 801 is specifically used for:

[0128] The x-coordinate of the ground positioning point in the scene image is obtained by using the x-coordinate of the center point of the annotation box, the length of the annotation box in the x-axis direction in the annotation box size, and the x-coordinate ratio. The y-coordinate of the ground positioning point in the scene image is obtained by using the y-coordinate of the center point of the annotation box, the length of the annotation box in the y-axis direction in the annotation box size, and the y-coordinate ratio.

[0129] In one embodiment, the above apparatus further includes: a mapping construction module, configured to:

[0130] Obtain the original point cloud set and the original scene image of the target scene; extract the ground point cloud from the original point cloud set and expand the ground point cloud to obtain the expanded ground point cloud; generate a coordinate mapping relationship based on the expanded ground point cloud and the original scene image.

[0131] In one embodiment, the mapping building module is specifically used for:

[0132] Based on the calibration relationship between the lidar and the camera equipment, the augmented ground points are converted into the original scene image to obtain the converted pixels of the augmented ground point cloud in the original scene image; the converted pixels are matched with the two-dimensional pixels in the original scene image; the successfully matched two-dimensional pixels are associated one-to-one with the corresponding augmented ground point cloud to obtain the coordinate mapping relationship.

[0133] In one embodiment, the above-described apparatus further includes: a model generation module, configured to:

[0134] Obtain labeled data for multiple sample images; input the labeled data of each sample image into the initial reference neural network model to obtain the initial reference ratio output by the initial reference neural network model; compare the corresponding initial reference ratio with the target reference ratio to obtain the comparison result; adjust the model parameters in the initial reference neural network model according to the comparison result of each sample image until the model converges to obtain the reference neural network model.

[0135] Each module in the aforementioned data mapping device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0136] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 9 As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a data mapping method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0137] Those skilled in the art will understand that Figure 9The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0138] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0139] Based on the target to be detected in the scene image of the target scene, obtain the two-dimensional coordinates of the ground positioning point of the target to be detected in the scene image; according to the preset coordinate mapping relationship, map the two-dimensional coordinates of the ground positioning point of the target to be detected into three-dimensional coordinates; wherein, the coordinate mapping relationship is established based on the two-dimensional pixels of the target scene and the expanded ground point cloud.

[0140] In one embodiment, the processor further performs the following steps when executing the computer program:

[0141] Obtain the annotation data of the target to be detected in the scene image; based on the annotation data, determine the two-dimensional coordinates of the ground positioning point of the target to be detected in the scene image.

[0142] In one embodiment, the annotation data includes the coordinates of the center point of the annotation box and the size of the annotation box; when the processor executes the computer program, it also performs the following steps:

[0143] The coordinates of the center point of the annotation box and the size of the annotation box are input into the reference neural network model to obtain the reference ratio output by the reference neural network model. The reference ratio represents the degree of positional deviation between the center point of the near-ground border of the annotation box and the ground positioning point of the target to be detected. Based on the coordinates of the center point of the annotation box, the size of the annotation box and the reference ratio, the two-dimensional coordinates of the ground positioning point of the target to be detected in the scene image are determined.

[0144] In one embodiment, the reference ratio includes a horizontal axis ratio and a vertical axis ratio; the processor, when executing the computer program, also performs the following steps:

[0145] The x-coordinate of the ground positioning point in the scene image is obtained by using the x-coordinate of the center point of the annotation box, the length of the annotation box in the x-axis direction in the annotation box size, and the x-coordinate ratio. The y-coordinate of the ground positioning point in the scene image is obtained by using the y-coordinate of the center point of the annotation box, the length of the annotation box in the y-axis direction in the annotation box size, and the y-coordinate ratio.

[0146] In one embodiment, the processor further performs the following steps when executing the computer program:

[0147] Obtain the original point cloud set and the original scene image of the target scene; extract the ground point cloud from the original point cloud set and expand the ground point cloud to obtain the expanded ground point cloud; generate a coordinate mapping relationship based on the expanded ground point cloud and the original scene image.

[0148] In one embodiment, the processor further performs the following steps when executing the computer program:

[0149] Based on the calibration relationship between the lidar and the camera equipment, the augmented ground points are converted into the original scene image to obtain the converted pixels of the augmented ground point cloud in the original scene image; the converted pixels are matched with the two-dimensional pixels in the original scene image; the successfully matched two-dimensional pixels are associated one-to-one with the corresponding augmented ground point cloud to obtain the coordinate mapping relationship.

[0150] In one embodiment, the processor further performs the following steps when executing the computer program:

[0151] Obtain labeled data for multiple sample images; input the labeled data of each sample image into the initial reference neural network model to obtain the initial reference ratio output by the initial reference neural network model; compare the corresponding initial reference ratio with the target reference ratio to obtain the comparison result; adjust the model parameters in the initial reference neural network model according to the comparison result of each sample image until the model converges to obtain the reference neural network model.

[0152] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0153] Based on the target to be detected in the scene image of the target scene, obtain the two-dimensional coordinates of the ground positioning point of the target to be detected in the scene image; according to the preset coordinate mapping relationship, map the two-dimensional coordinates of the ground positioning point of the target to be detected into three-dimensional coordinates; wherein, the coordinate mapping relationship is established based on the two-dimensional pixels of the target scene and the expanded ground point cloud.

[0154] In one embodiment, when the computer program is executed by the processor, it further performs the following steps:

[0155] Obtain the annotation data of the target to be detected in the scene image; based on the annotation data, determine the two-dimensional coordinates of the ground positioning point of the target to be detected in the scene image.

[0156] In one embodiment, the annotation data includes the coordinates of the center point of the annotation box and the size of the annotation box; when the computer program is executed by the processor, it also performs the following steps:

[0157] The coordinates of the center point of the annotation box and the size of the annotation box are input into the reference neural network model to obtain the reference ratio output by the reference neural network model. The reference ratio represents the degree of positional deviation between the center point of the annotation box and the ground positioning point of the target to be detected. Based on the coordinates of the center point of the annotation box, the size of the annotation box and the reference ratio, the two-dimensional coordinates of the ground positioning point of the target to be detected in the scene image are determined.

[0158] In one embodiment, the reference ratio includes a ratio of the horizontal axis and a ratio of the vertical axis; when the computer program is executed by the processor, it also performs the following steps:

[0159] The x-coordinate of the ground positioning point in the scene image is obtained by using the x-coordinate of the center point of the annotation box, the length of the annotation box in the x-axis direction in the annotation box size, and the x-coordinate ratio. The y-coordinate of the ground positioning point in the scene image is obtained by using the y-coordinate of the center point of the annotation box, the length of the annotation box in the y-axis direction in the annotation box size, and the y-coordinate ratio.

[0160] In one embodiment, when the computer program is executed by the processor, it further performs the following steps:

[0161] Obtain the original point cloud set and the original scene image of the target scene; extract the ground point cloud from the original point cloud set and expand the ground point cloud to obtain the expanded ground point cloud; generate a coordinate mapping relationship based on the expanded ground point cloud and the original scene image.

[0162] In one embodiment, when the computer program is executed by the processor, it further performs the following steps:

[0163] Based on the calibration relationship between the lidar and the camera equipment, the augmented ground points are converted into the original scene image to obtain the converted pixels of the augmented ground point cloud in the original scene image; the converted pixels are matched with the two-dimensional pixels in the original scene image; the successfully matched two-dimensional pixels are associated one-to-one with the corresponding augmented ground point cloud to obtain the coordinate mapping relationship.

[0164] In one embodiment, when the computer program is executed by the processor, it further performs the following steps:

[0165] Obtain labeled data for multiple sample images; input the labeled data of each sample image into the initial reference neural network model to obtain the initial reference ratio output by the initial reference neural network model; compare the corresponding initial reference ratio with the target reference ratio to obtain the comparison result; adjust the model parameters in the initial reference neural network model according to the comparison result of each sample image until the model converges to obtain the reference neural network model.

[0166] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0167] Based on the target to be detected in the scene image of the target scene, obtain the two-dimensional coordinates of the ground positioning point of the target to be detected in the scene image; according to the preset coordinate mapping relationship, map the two-dimensional coordinates of the ground positioning point of the target to be detected into three-dimensional coordinates; wherein, the coordinate mapping relationship is established based on the two-dimensional pixels of the target scene and the expanded ground point cloud.

[0168] In one embodiment, when the computer program is executed by the processor, it further performs the following steps:

[0169] Obtain the annotation data of the target to be detected in the scene image; based on the annotation data, determine the two-dimensional coordinates of the ground positioning point of the target to be detected in the scene image.

[0170] In one embodiment, the annotation data includes the coordinates of the center point of the annotation box and the size of the annotation box; when the computer program is executed by the processor, it also performs the following steps:

[0171] The coordinates of the center point of the annotation box and the size of the annotation box are input into the reference neural network model to obtain the reference ratio output by the reference neural network model. The reference ratio represents the degree of positional deviation between the center point of the near-ground border of the annotation box and the ground positioning point of the target to be detected. Based on the coordinates of the center point of the annotation box, the size of the annotation box and the reference ratio, the two-dimensional coordinates of the ground positioning point of the target to be detected in the scene image are determined.

[0172] In one embodiment, the reference ratio includes a ratio of the horizontal axis and a ratio of the vertical axis; when the computer program is executed by the processor, it also performs the following steps:

[0173] The x-coordinate of the ground positioning point in the scene image is obtained by using the x-coordinate of the center point of the annotation box, the length of the annotation box in the x-axis direction in the annotation box size, and the x-coordinate ratio. The y-coordinate of the ground positioning point in the scene image is obtained by using the y-coordinate of the center point of the annotation box, the length of the annotation box in the y-axis direction in the annotation box size, and the y-coordinate ratio.

[0174] In one embodiment, when the computer program is executed by the processor, it further performs the following steps:

[0175] Obtain the original point cloud set and the original scene image of the target scene; extract the ground point cloud from the original point cloud set and expand the ground point cloud to obtain the expanded ground point cloud; generate a coordinate mapping relationship based on the expanded ground point cloud and the original scene image.

[0176] In one embodiment, when the computer program is executed by the processor, it further performs the following steps:

[0177] Based on the calibration relationship between the lidar and the camera equipment, the augmented ground points are converted into the original scene image to obtain the converted pixels of the augmented ground point cloud in the original scene image; the converted pixels are matched with the two-dimensional pixels in the original scene image; the successfully matched two-dimensional pixels are associated one-to-one with the corresponding augmented ground point cloud to obtain the coordinate mapping relationship.

[0178] In one embodiment, when the computer program is executed by the processor, it further performs the following steps:

[0179] Obtain labeled data for multiple sample images; input the labeled data of each sample image into the initial reference neural network model to obtain the initial reference ratio output by the initial reference neural network model; compare the corresponding initial reference ratio with the target reference ratio to obtain the comparison result; adjust the model parameters in the initial reference neural network model according to the comparison result of each sample image until the model converges to obtain the reference neural network model.

[0180] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0181] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0182] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data mapping method, characterized in that, The method includes: Obtain the annotation data of the target to be detected in the scene image; the annotation data includes the coordinates of the center point of the annotation box and the size of the annotation box. The coordinates of the center point of the annotation box and the size of the annotation box are input into the reference neural network model to obtain the reference ratio output by the reference neural network model; the reference ratio characterizes the degree of positional deviation between the center point of the annotation box and the ground positioning point of the target to be detected; Based on the coordinates of the center point of the annotation box, the size of the annotation box, and the reference ratio, the two-dimensional coordinates of the ground positioning point of the target to be detected in the scene image are determined; According to a preset coordinate mapping relationship, the two-dimensional coordinates of the ground positioning point of the target to be detected are mapped to three-dimensional coordinates; wherein, the coordinate mapping relationship is established based on the two-dimensional pixels of the target scene and the expanded ground point cloud.

2. The method according to claim 1, characterized in that, The reference ratio includes the ratio of the horizontal axis and the ratio of the vertical axis; The step of determining the two-dimensional coordinates of the ground positioning point of the target to be detected in the scene image based on the coordinates of the center point of the annotation box, the size of the annotation box, and the reference ratio includes: The horizontal coordinate of the ground positioning point in the scene image is obtained based on the horizontal coordinate of the center point of the annotation box, the length of the annotation box in the horizontal direction of the annotation box size, and the horizontal coordinate ratio. The ordinate of the ground positioning point in the scene image is obtained based on the ordinate of the center point coordinate of the annotation box, the length of the annotation box in the ordinate direction in the annotation box size, and the ordinate ratio.

3. The method according to claim 1 or 2, characterized in that, The method further includes: Obtain the original point cloud set and original scene image of the target scene; Extract the ground point cloud from the original point cloud set, and expand the ground point cloud to obtain the expanded ground point cloud; The coordinate mapping relationship is generated based on the expanded ground point cloud and the original scene image.

4. The method according to claim 3, characterized in that, The step of generating the coordinate mapping relationship based on the expanded ground point cloud and the original scene image includes: Based on the calibration relationship between the lidar and the camera equipment, the expanded ground points are converted into the original scene image to obtain the converted pixel points of the expanded ground point cloud in the original scene image; The converted pixels are matched with the two-dimensional pixels in the original scene image; The successfully matched two-dimensional pixels are associated one-to-one with the corresponding expanded ground point cloud to obtain the coordinate mapping relationship.

5. The method according to claim 1 or 2, characterized in that, The method further includes: Obtain labeled data for multiple sample images; input the labeled data of each sample image into an initial reference neural network model to obtain the initial reference ratio output by the initial reference neural network model; Compare the corresponding initial reference ratio with the target reference ratio to obtain the comparison result; The model parameters in the initial reference neural network model are adjusted according to the comparison results of each sample image until the model converges, thus obtaining the reference neural network model.

6. A data mapping device, characterized in that, The device includes: A two-dimensional coordinate determination module is used to: acquire annotation data of the target to be detected in a scene image; the annotation data includes the coordinates of the center point of the annotation box and the size of the annotation box; input the coordinates of the center point of the annotation box and the size of the annotation box into a reference neural network model to obtain a reference ratio output by the reference neural network model; the reference ratio characterizes the degree of positional deviation between the center point of the annotation box and the ground positioning point of the target to be detected; and determine the two-dimensional coordinates of the ground positioning point of the target to be detected in the scene image based on the coordinates of the center point of the annotation box, the size of the annotation box, and the reference ratio. The three-dimensional coordinate determination module is used to map the two-dimensional coordinates of the ground positioning point of the target to be detected into three-dimensional coordinates according to a preset coordinate mapping relationship; wherein the coordinate mapping relationship is established based on the two-dimensional pixels of the target scene and the expanded ground point cloud.

7. The apparatus according to claim 6, characterized in that, The reference ratio includes the ratio of the horizontal axis and the ratio of the vertical axis; The two-dimensional coordinate determination module is further configured to: obtain the abscissa of the ground positioning point in the scene image based on the abscissa of the center point coordinate of the annotation box, the length of the annotation box in the abscissa direction in the annotation box size, and the abscissa ratio; and obtain the ordinate of the ground positioning point in the scene image based on the ordinate of the center point coordinate of the annotation box, the length of the annotation box in the ordinate direction in the annotation box size, and the ordinate ratio.

8. The apparatus according to claim 6 or 7, characterized in that, The device further includes: The mapping construction module is used to obtain the original point cloud set and the original scene image under the target scene; extract the ground point cloud from the original point cloud set, and expand the ground point cloud to obtain the expanded ground point cloud; generate the coordinate mapping relationship based on the expanded ground point cloud and the original scene image.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images