A training and matching method, device and medium for a cross-modal matching model

Abstract

The present application provides a training, matching method, device and medium for a cross-modal matching model, and relates to the field of image recognition technology. Obtain multiple groups of sample cross-modal sample data; use a preset visual branch network to detect each group of cross-modal sample data, and obtain sample visual detection results corresponding to each group of cross-modal sample data; use a preset point cloud branch network to detect each group of cross-modal sample data, and obtain sample point cloud detection results corresponding to each group of cross-modal sample data; calculate the target detection loss value based on the first three-dimensional detection frame, the first three-dimensional annotation frame of the target object in the sample roadside image, the second three-dimensional detection frame, and the second three-dimensional annotation frame of the target object in the corresponding sample roadside point cloud data; calculate the target loss function value based on the target detection loss value; adjust the preset visual branch network and the preset point cloud branch network to obtain a cross-modal matching model. Thereby, the training efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a training and matching method, device and medium for a cross-modal matching model. Background Art

[0002] With the rapid development of deep learning technology, the development potential of computers in more and more fields has been tapped. Scene recognition is a basic task for many applications, such as autonomous driving, autonomous robots, and augmented reality.

[0003] Point cloud data is a kind of three-dimensional data information of the space of the external environment that is actively acquired by a laser radar scanner by emitting lasers and receiving reflected lasers. It is sparse, discrete, and unstructured. In the field of point clouds, similar to the development in the field of images, traditional point cloud semantic extraction methods are designed as hand-designed features for point cloud descriptors, which are not learnable and thus have poor adaptability to specific tasks, and the results are difficult to meet actual needs.

[0004] Cross-modal fusion is the integration of multiple types of data to enhance feature expression and improve model performance, especially to overcome the inherent defects caused by single-modal data. Among the existing cross-modal fusion technologies, one is that the point cloud branch and the visual branch are coupled for detection. The 3D object proposals of the visual branch come from the radar branch. In actual use, the point cloud and image data must be input into the model at the same time. The point cloud cannot be removed (for example, the lidar is offline, not running, or the configuration fails). During the reasoning process, the two branches cannot work independently to output object features, resulting in low data processing efficiency. Summary of the invention

[0005] The purpose of the present invention is to address the deficiencies in the above-mentioned prior art and provide a method and device for locating problems in a front-end task node component to solve the problems of low data processing efficiency in the prior art.

[0006] To achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

[0007] In a first aspect, an embodiment of the present application provides a method for training a cross-modal matching model, wherein the cross-modal matching model includes: a visual branch network and a point cloud branch network, and the method includes:

[0008] Acquire multiple groups of sample cross-modal sample data, each group of cross-modal sample data includes: a sample roadside image and corresponding sample roadside point cloud data;

[0009] Using a preset visual branch network, the sample roadside image in each group of cross-modal sample data is detected to obtain a sample visual detection result corresponding to each group of cross-modal sample data, wherein the sample visual detection result includes: a first three-dimensional detection frame of the target object;

[0010] Using a preset point cloud branch network, the sample roadside point cloud data in each group of cross-modal sample data is detected to obtain a sample point cloud detection result corresponding to each group of cross-modal sample data, wherein the sample point cloud detection result includes: a second three-dimensional detection frame of the target object;

[0011] Calculating a target detection loss value according to the first three-dimensional detection frame, the first three-dimensional annotation frame of the target object in the sample roadside image, the second three-dimensional detection frame, and the corresponding second three-dimensional annotation frame of the target object in the sample roadside point cloud data;

[0012] Calculating a target loss function value according to the target detection loss value;

[0013] The preset visual branch network and the preset point cloud branch network are adjusted to obtain the cross-modal matching model.

[0014] Optionally, the calculating the target detection loss value according to the first three-dimensional detection frame, the first three-dimensional annotation frame of the target object in the sample roadside image, the second three-dimensional detection frame, and the corresponding second three-dimensional annotation frame of the target object in the sample roadside point cloud data includes:

[0015] Calculating a first detection loss value according to the first three-dimensional detection frame and a first three-dimensional annotation frame of the target object in the sample roadside image;

[0016] Calculating a second detection loss value according to the second three-dimensional detection frame and the corresponding second three-dimensional annotation frame of the target object in the sample roadside point cloud data;

[0017] The target detection loss value is calculated according to the first detection loss value and the second detection loss value.

[0018] Optionally, the visual detection result further includes: first bird's-eye view depth description information; the point cloud detection result further includes: second bird's-eye view depth description information;

[0019] Before calculating the target loss function value according to the target detection loss value, the method further includes:

[0020] Calculating a depth description loss value according to the first top-view depth description information and the second top-view depth description information;

[0021] The calculating the target loss function value according to the target detection loss value includes:

[0022] The target loss function value is calculated according to the target detection loss value and the depth description loss value.

[0023] Optionally, the cross-modal matching model further includes: a deep visual decoder;

[0024] Before calculating the target loss function value according to the target detection loss value and the depth description loss value, the method further includes:

[0025] Generate a first depth map and a second depth map using the depth vision decoder according to the first top-view depth description information and the second top-view depth description information;

[0026] Calculating a depth estimation loss value according to the first depth map and the second depth map;

[0027] The calculating the target loss function value according to the target detection loss value and the depth description loss value includes:

[0028] The target loss function value is calculated according to the target detection loss value, the depth description loss and the depth estimation loss value.

[0029] Optionally, the cross-modal matching model further includes: an image decoder; the visual detection result further includes: overhead image description information; before calculating the target loss function value according to the target detection loss value, the depth description loss and the depth estimation loss value, the method further includes:

[0030] According to the overhead image description information, using the image decoder, generating a predicted color image corresponding to the sample roadside image;

[0031] Generate a first visual branch cutout according to the first three-dimensional detection frame and the sample roadside image;

[0032] Generating a second visual branch cutout according to the first three-dimensional detection frame and the predicted color image;

[0033] Calculating a visual visualization loss value according to the first visual branch cutout and the second visual branch cutout;

[0034] The calculating the target loss function value according to the target detection loss value, the depth description loss and the depth estimation loss value includes:

[0035] The target loss function value is calculated according to the target detection loss value, the depth description loss, the depth estimation loss value and the visual visualization loss value.

[0036] Optionally, generating a first visual branch cutout according to the first three-dimensional detection frame and the sample roadside image includes:

[0037] Generate the first visual branch cutout according to the first three-dimensional detection frame, the first depth map, and the sample roadside image;

[0038] The step of generating a second visual branch cutout according to the first three-dimensional detection frame and the predicted color image includes:

[0039] Generate the second visual branch cutout according to the first three-dimensional detection box, the first depth map and the predicted color image.

[0040] Optionally, before calculating the target loss function value according to the target detection loss value, the depth description loss and the depth estimation loss value, the method further includes:

[0041] Generate a point cloud branch cutout according to the second three-dimensional detection frame and the sample roadside point cloud data;

[0042] Calculating a point cloud visualization loss value according to the point cloud branch cutout and the second depth map;

[0043] The calculating the target loss function value according to the target detection loss value, the depth description loss and the depth estimation loss value includes:

[0044] The target loss function value is calculated according to the target detection loss value, the depth description loss, the depth estimation loss value and the point cloud visualization loss value.

[0045] In a second aspect, an embodiment of the present application provides a target matching method, the method comprising:

[0046] Obtain target roadside image and corresponding target roadside point cloud data;

[0047] The visual branch network and the point cloud branch network in the cross-modal matching model are used to process the target roadside image and the target roadside point cloud data respectively to obtain a visual detection result and a point cloud detection result, wherein the visual detection result includes: a plurality of first detection frames, and the point cloud detection result includes: a plurality of second detection frames; wherein the cross-modal matching model is a model trained by the method described in any one of the first aspects above;

[0048] Object matching is performed on the multiple first detection frames and the multiple second detection frames.

[0049] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium through the bus, and the processor executes the machine-readable instructions to perform the steps of the training method of the cross-modal matching model as described in any one of the first aspects, or the steps of the target matching method as described in the second aspect.

[0050] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the training method of the cross-modal matching model as described in any one of the first aspects, or the steps of the target matching method as described in the second aspect are executed.

[0051] Compared with the prior art, this application has the following beneficial effects:

[0052] The present application provides a training and matching method, device and medium of a cross-modal matching model. The cross-modal matching model includes: a visual branch network and a point cloud branch network, and obtains multiple groups of sample cross-modal sample data. Each group of cross-modal sample data includes: a sample roadside image and corresponding sample roadside point cloud data; a preset visual branch network is used to detect the sample roadside image in each group of cross-modal sample data to obtain a sample visual detection result corresponding to each group of cross-modal sample data, and the sample visual detection result includes: a first three-dimensional detection frame of the target object; a preset point cloud branch network is used to detect each sample roadside image in each group of cross-modal sample data. The sample roadside point cloud data in the group of cross-modal sample data are detected to obtain the sample point cloud detection results corresponding to each group of cross-modal sample data, and the sample point cloud detection results include: the second three-dimensional detection frame of the target object; the target detection loss value is calculated according to the first three-dimensional detection frame, the first three-dimensional annotation frame of the target object in the sample roadside image, the second three-dimensional detection frame and the second three-dimensional annotation frame of the target object in the corresponding sample roadside point cloud data; the target loss function value is calculated according to the target detection loss value; the preset visual branch network and the preset point cloud branch network are adjusted to obtain a cross-modal matching model. Therefore, the preset visual branch network and the preset point cloud branch network are independent of each other, and there is no order of precedence. Multiple groups of cross-modal sample data can be input at the same time for training and matching, which improves the training efficiency. Through cross-modal joint training, the training accuracy is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A schematic diagram of a flow chart of a training method for a cross-modal matching model provided in this application;

[0054] Figure 2 A flowchart of a method for calculating a target detection loss value provided in an embodiment of the present application;

[0055] Figure 3 A flowchart of a method for calculating a target loss function value provided in an embodiment of the present application;

[0056] Figure 4 A flowchart of another method for calculating the target loss function value provided in an embodiment of the present application;

[0057] Figure 5 A flowchart of another method for calculating a target loss function value provided in an embodiment of the present application;

[0058] Figure 6 A flowchart of a method for generating a branch cutout provided in an embodiment of the present application;

[0059] Figure 7 A flowchart of a method for calculating a target loss function value provided in an embodiment of the present application;

[0060] Figure 8 A flowchart of a target matching method is shown;

[0061] Fig. 9 A structural diagram of a cross-modal matching model provided in an embodiment of the present application;

[0062] Fig.10 A schematic diagram of a cross-modal matching model training device provided in an embodiment of the present application;

[0063] Fig.11 A schematic diagram of a target matching device provided in an embodiment of the present application;

[0064] Fig.12 A schematic diagram of an electronic device provided in an embodiment of the present application.

[0065] Icons: 1001 - first acquisition module, 1002 - first detection module, 1003 - second detection module, 1004 - first calculation module, 1005 - second calculation module, 1006 - adjustment module, 1101 - second acquisition module, 1102 - third detection module, 1103 - matching module, 1201 - processor, 1202 - storage medium. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical scheme and advantages of the embodiments of the present invention clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0067] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for protection, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0068] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0069] In addition, the terms “first”, “second”, etc., if used, are merely used to distinguish between the descriptions and should not be understood as indicating or implying relative importance.

[0070] It should be noted that, in the absence of conflict, the features in the embodiments of the present invention may be combined with each other.

[0071] The following is an explanation of a cross-modal matching model training method provided in this application through a specific example. Figure 1 The present invention provides a flow chart of a training method for a cross-modal matching model, wherein the cross-modal matching model includes: a visual branch network and a point cloud branch network, and the execution subject of the method is an electronic device having a computing and processing function. Figure 1 As shown, the method includes:

[0072] S101. Obtain multiple groups of cross-modal sample data.

[0073] Each set of cross-modal sample data includes: a sample roadside image and corresponding sample roadside point cloud data;

[0074] For example, road test equipment (roadside camera equipment, roadside radar equipment) is used to collect sample roadside images and sample roadside point cloud data of multiple highway scenes, and the sample roadside images and sample roadside point cloud data of the same highway scene are combined into a set of cross-modal sample data.

[0075] The same set of cross-modal sample data are matched. If the same set of cross-modal sample data obtained by model training are also matched, it means that the model has been trained.

[0076] S102: Using a preset visual branch network, detect the sample roadside images in each group of cross-modal sample data to obtain sample visual detection results corresponding to each group of cross-modal sample data.

[0077] The sample visual detection result includes: a first three-dimensional detection frame of the target object.

[0078] For example, there may be multiple objects to be detected in a highway scene (e.g., multiple vehicles), and there are correspondingly multiple objects in the sample roadside image. By detecting the sample roadside image, a first three-dimensional detection frame of each object is obtained, that is, the first three-dimensional detection frame of the target object.

[0079] S103: Using a preset point cloud branch network, the sample roadside point cloud data in each group of cross-modal sample data is detected to obtain a sample point cloud detection result corresponding to each group of cross-modal sample data.

[0080] The sample point cloud detection results include: a second three-dimensional detection frame of the target object.

[0081] For example, there are multiple objects in the sample roadside point cloud data. By detecting the sample roadside point cloud data, a second three-dimensional detection frame of each object is obtained, that is, a second three-dimensional detection frame of the target object.

[0082] S104, calculating a target detection loss value according to the first three-dimensional detection frame, the first three-dimensional annotation frame of the target object in the sample roadside image, the second three-dimensional detection frame, and the second three-dimensional annotation frame of the target object in the corresponding sample roadside point cloud data.

[0083] The first 3D detection frame, the first 3D annotation frame, the second 3D detection frame, and the second 3D annotation frame all represent the 3D frames of the target object. The more similar the above four 3D frames are, the smaller the target detection loss value is.

[0084] S105. Calculate the target loss function value according to the target detection loss value.

[0085] For example, the target detection loss value may be determined as the target loss function value.

[0086] S106. Adjust parameters of a preset visual branch network and a preset point cloud branch network to obtain a cross-modal matching model.

[0087] For example, if the target loss function value does not meet the preset model training cutoff condition, the preset visual branch network and the preset point cloud branch network are adjusted until the target loss function value meets the preset model training cutoff condition, and the preset visual branch network and the preset point cloud branch network after parameter adjustment are used as cross-modal matching models.

[0088] The preset visual branch network and the preset point cloud branch network of the present application are independent of each other and there is no order of precedence. Multiple groups of cross-modal sample data can be input simultaneously for training and matching, which improves the training efficiency. Through cross-modal joint training, the training accuracy is improved.

[0089] In summary, in this embodiment, the cross-modal matching model includes: a visual branch network and a point cloud branch network, and multiple groups of sample cross-modal sample data are obtained, each group of cross-modal sample data includes: a sample roadside image, and corresponding sample roadside point cloud data; a preset visual branch network is used to detect the sample roadside image in each group of cross-modal sample data, and a sample visual detection result corresponding to each group of cross-modal sample data is obtained, and the sample visual detection result includes: a first three-dimensional detection frame of the target object; a preset point cloud branch network is used to detect the sample roadside point cloud data in each group of cross-modal sample data, and a sample point cloud detection result corresponding to each group of cross-modal sample data is obtained, and the sample point cloud detection result includes: a second three-dimensional detection frame of the target object; according to the first three-dimensional detection frame, the first three-dimensional annotation frame of the target object in the sample roadside image, the second three-dimensional detection frame and the second three-dimensional annotation frame of the target object in the corresponding sample roadside point cloud data, the target detection loss value is calculated; according to the target detection loss value, the target loss function value is calculated; the preset visual branch network and the preset point cloud branch network are adjusted to obtain a cross-modal matching model. Therefore, the preset visual branch network and the preset point cloud branch network are independent of each other and there is no order of precedence. Multiple groups of cross-modal sample data can be input simultaneously for training and matching, which improves the training efficiency. Through cross-modal joint training, the training accuracy is improved.

[0090] In the above Figure 1 Based on the corresponding embodiment, the embodiment of the present application also provides a method for calculating the target detection loss value. Figure 2 A flow chart of a method for calculating target detection loss value provided in an embodiment of the present application. Figure 2 As shown, in S104, the target detection loss value is calculated according to the first three-dimensional detection frame, the first three-dimensional annotation frame of the target object in the sample roadside image, the second three-dimensional detection frame, and the second three-dimensional annotation frame of the target object in the corresponding sample roadside point cloud data, including:

[0091] S201. Calculate a first detection loss value according to a first three-dimensional detection frame and a first three-dimensional annotation frame of a target object in a sample roadside image.

[0092] The first 3D annotation frame of the target object in the sample roadside image represents the real 3D frame, and the first 3D detection frame represents the predicted 3D frame. The first detection loss value can be calculated through the real 3D frame and the predicted 3D frame.

[0093] For example, a 3D target detection loss calculation method may be used to calculate the first detection loss value.

[0094] S202: Calculate a second detection loss value according to the second three-dimensional detection frame and the corresponding second three-dimensional annotation frame of the target object in the sample roadside point cloud data.

[0095] The second 3D annotation frame of the target object in the sample roadside point cloud data represents the real 3D frame, and the second 3D detection frame represents the predicted 3D frame. The second detection loss value can be calculated through the real 3D frame and the predicted 3D frame.

[0096] For example, a 3D target detection loss calculation method may be used to calculate the second detection loss value.

[0097] S203. Calculate a target detection loss value according to the first detection loss value and the second detection loss value.

[0098] The calculation method of the target detection loss value is shown in the following formula (1):

[0099] loss_proposal=Wp1*loss_proposal_cam + Wp1*loss_proposal_ pc (1)

[0100] Among them, loss_proposal is the target detection loss value, loss_proposal_cam is the first detection loss value, Wp1 is the weight of the first detection loss value, loss_proposal_pc is the second detection loss value, and Wp1 is the weight of the second detection loss value.

[0101] In summary, in this embodiment, the first detection loss value is calculated based on the first three-dimensional detection frame and the first three-dimensional annotation frame of the target object in the sample roadside image; the second detection loss value is calculated based on the second three-dimensional detection frame and the second three-dimensional annotation frame of the target object in the corresponding sample roadside point cloud data; and the target detection loss value is calculated based on the first detection loss value and the second detection loss value. Thus, the target detection loss value is accurately calculated.

[0102] In the above Figure 1 Based on the corresponding embodiments, the embodiments of the present application also provide a method for calculating the target loss function value. Figure 3 A flowchart of a method for calculating a target loss function value provided in an embodiment of the present application, wherein the visual detection result also includes: first overhead depth description information; the point cloud detection result also includes: second overhead depth description information.

[0103] like Figure 3As shown, before calculating the target loss function value according to the target detection loss value in S105, the method further includes:

[0104] S301. Calculate a depth description loss value according to first bird's-eye view depth description information and second bird's-eye view depth description information.

[0105] The first bird's-eye view depth description information and the second bird's-eye view depth description information in the same set of cross-modal sample data are used to calculate the depth description loss value, so that the detection results of the visual branch and the point cloud branch can be compared.

[0106] The calculation method of the depth description loss value is shown in the following formula (2):

[0107] loss_desc_depth=1-similarity_desc_depth (2)

[0108] Among them, loss_desc_depth is the depth description loss value, similarity_desc_depth is the similarity between the first top-view depth description information and the second top-view depth description information, and the similarity calculation method can be the Euclidean distance method or the cosine distance method, which is not limited in this application.

[0109] In S105, the target loss function value is calculated according to the target detection loss value, including:

[0110] S302. Calculate the target loss function value according to the target detection loss value and the depth description loss value.

[0111] The specific calculation method of the target loss function value can also be shown in the following formula (3):

[0112] Loss=w1*loss_proposal+w2*loss_desc_depth (3)

[0113] Among them, w2 is the weight of the visual visualization loss value.

[0114] In summary, in this embodiment, the visual detection result also includes: the first top-view depth description information; the point cloud detection result also includes: the second top-view depth description information; the depth description loss value is calculated based on the first top-view depth description information and the second top-view depth description information; the target loss function value is calculated based on the target detection loss value and the depth description loss value. Therefore, the introduction of the depth description loss value makes the calculation of the target loss function value more accurate.

[0115] In the above Figure 3 Based on the corresponding embodiment, the embodiment of the present application also provides another method for calculating the target loss function value. Figure 4A flowchart of another method for calculating a target loss function value provided in an embodiment of the present application, wherein the cross-modal matching model further includes: a deep visual decoder.

[0116] like Figure 4 As shown, before calculating the target loss function value according to the target detection loss value and the depth description loss value in S302, the method further includes:

[0117] S401 . Generate a first depth map and a second depth map using a depth vision decoder according to first bird's-eye view depth description information and second bird's-eye view depth description information.

[0118] The first top-view depth description information is input into the depth vision decoder to generate a first depth map; the second top-view depth description information is input into the depth vision decoder to generate a second depth map.

[0119] The first depth map and the second depth map are single-channel two-dimensional depth maps.

[0120] S402: Calculate a depth estimation loss value according to the first depth map and the second depth map.

[0121] The calculation method of the depth estimation loss value is shown in the following formula (4):

[0122] loss_depth=DepthCriterion(depth_pc, depth_cam) (4)

[0123] Among them, loss_depth is the depth estimation loss value, depth_cam is the first depth map, and depth_pc is the second depth map.

[0124] In S302, the target loss function value is calculated according to the target detection loss value and the depth description loss value, including:

[0125] S403. Calculate the target loss function value according to the target detection loss value, the depth description loss and the depth estimation loss value.

[0126] The specific calculation method of the target loss function value can also be shown in the following formula (5):

[0127] Loss=w1*loss_proposal+w2*loss_desc_depth+w3*loss_depth (5)

[0128] Among them, w3 is the weight of the visual visualization loss value.

[0129] In summary, in this embodiment, the cross-modal matching model also includes: a depth vision decoder; according to the first top-view depth description information and the second top-view depth description information, a depth vision decoder is used to generate a first depth map and a second depth map; according to the first depth map and the second depth map, a depth estimation loss value is calculated; according to the target detection loss value, the depth description loss and the depth estimation loss value, a target loss function value is calculated. Therefore, the introduction of the depth estimation loss value makes the calculation of the target loss function value more accurate.

[0130] In the above Figure 4 Based on the corresponding embodiment, the embodiment of the present application also provides another method for calculating the target loss function value. Figure 5 A flowchart of another method for calculating the target loss function value provided in an embodiment of the present application, wherein the cross-modal matching model further includes: an image decoder; and the visual detection result further includes: overhead image description information. By introducing two descriptors, visualization of the descriptors is achieved, the interpretability of the deep learning network is enhanced, and the internal structure and process of the model are made clearer.

[0131] like Figure 5 As shown, before calculating the target loss function value according to the target detection loss value, the depth description loss and the depth estimation loss value in S403, the method further includes:

[0132] S501 : Generate a predicted color image corresponding to a sample roadside image using an image decoder according to the bird's-eye view image description information.

[0133] The top view image description information is input into the image decoder to obtain a three-channel RGB image with a shape of (C=3, H, W), where HW is a custom image size, such as 36x36, 72x72, etc.

[0134] The three-channel RGB image is a predicted color image corresponding to the sample roadside image.

[0135] S502: Generate a first visual branch cutout according to the first three-dimensional detection frame and the sample roadside image.

[0136] In the same three-dimensional reference coordinate system, the color information of the sample roadside image is filled into the corresponding position in the first three-dimensional detection frame to obtain the first visual branch cutout. For example, the calculation can be performed based on preset encoder parameters and inverse transmission transform IPM.

[0137] S503: Generate a second visual branch cutout according to the first three-dimensional detection frame and the predicted color image.

[0138] In the same three-dimensional reference coordinate system, the color information of the predicted color image is filled into the corresponding position in the first three-dimensional detection frame to obtain the second visual branch cutout. For example, the calculation can be performed based on preset encoder parameters and inverse transmission transform IPM.

[0139] S504: Calculate a visual visualization loss value according to the first visual branch cutout and the second visual branch cutout.

[0140] The specific calculation method of the visual visualization loss value is shown in the following formula (6):

[0141] loss_vis_cam=ImageCriterion(capture_image,capture_vis) (6)

[0142] Among them, loss_vis_cam is the visual visualization loss value, capture_image is the first visual branch cutout, and capture_vis is the second visual branch cutout.

[0143] In S403, the target loss function value is calculated according to the target detection loss value, the depth description loss and the depth estimation loss value, including:

[0144] S505. Calculate the target loss function value according to the target detection loss value, the depth description loss value, the depth estimation loss value and the visual visualization loss value.

[0145] The specific calculation method of the target loss function value can also be shown in the following formula (7):

[0146] Loss=w1*loss_proposal+w2*loss_desc_depth+w3*loss_depth+

[0147] w4*loss_vis_cam(7)

[0148] Among them, w4 is the weight of the visual visualization loss value.

[0149] In summary, in this embodiment, the cross-modal matching model also includes: an image decoder; the visual detection result also includes: overhead image description information; according to the overhead image description information, the image decoder is used to generate a predicted color image corresponding to the sample roadside image; according to the first three-dimensional detection frame and the sample roadside image, a first visual branch cutout is generated; according to the first three-dimensional detection frame and the predicted color image, a second visual branch cutout is generated; according to the first visual branch cutout and the second visual branch cutout, a visual visualization loss value is calculated; according to the target detection loss value, the depth description loss value, the depth estimation loss value and the visual visualization loss value, the target loss function value is calculated. Thus, a target object image under a small range of viewing angle is presented, and the visual visualization loss value is introduced, so that the calculation of the target loss function value is more accurate.

[0150] In the above Figure 5 On the basis of the corresponding embodiment, the embodiment of the present application also provides a method for generating branch cutouts. Figure 6 A flow chart of a method for generating a branch cutout provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, in S502, generating a first visual branch cutout according to the first three-dimensional detection frame and the sample roadside image includes:

[0151] S601: Generate a first visual branch cutout according to a first three-dimensional detection frame, a first depth map, and a sample roadside image.

[0152] The introduction of the first depth map makes the generation of the first visual branch cutout more accurate and realistic.

[0153] In S503, generating a second visual branch cutout according to the first three-dimensional detection frame and the predicted color image includes:

[0154] S602: Generate a second visual branch cutout according to the first three-dimensional detection frame, the first depth map, and the predicted color image.

[0155] The introduction of the first depth map makes the generation of the second visual branch cutout more accurate and realistic.

[0156] In summary, in this embodiment, the first visual branch cutout is generated based on the first 3D detection frame, the first depth map and the sample roadside image; the second visual branch cutout is generated based on the first 3D detection frame, the first depth map and the predicted color image. Therefore, the introduction of the first depth map makes the generation of the visual branch cutout more accurate and realistic.

[0157] In the above Figure 4 Based on the corresponding embodiment, the embodiment of the present application also provides a method for calculating the target loss function value. Figure 7 A flowchart of a method for calculating a target loss function value is provided in an embodiment of the present application. Figure 7 As shown, before calculating the target loss function value according to the target detection loss value, the depth description loss and the depth estimation loss value in S403, the method further includes:

[0158] S701: Generate a point cloud branch cutout according to the second three-dimensional detection frame and sample roadside point cloud data.

[0159] In the same three-dimensional reference coordinate system, the sample roadside point cloud data is filled into the corresponding position in the second three-dimensional detection frame to obtain the point cloud branch cutout. For example, the calculation can be performed based on the preset encoder parameters and the inverse transmission transform IPM.

[0160] The point cloud branch is cut out to be a point cloud image of the target object.

[0161] S702: Calculate the point cloud visualization loss value according to the point cloud branch cutout and the second depth map.

[0162] The specific calculation method of the point cloud visualization loss value is shown in the following formula (8):

[0163] loss_vis_pc=PointCloudCriterion (capture_pc, depth_pc) (8)

[0164] Among them, loss_vis_pc is the point cloud visualization loss value, PointCloudCriterion is the point cloud visualization loss function, capture_pc is the point cloud branch cutout, and depth_pc is the second depth map.

[0165] In S403, the target loss function value is calculated according to the target detection loss value, the depth description loss and the depth estimation loss value, including:

[0166] S703. Calculate the target loss function value according to the target detection loss value, the depth description loss value, the depth estimation loss value and the point cloud visualization loss value.

[0167] The specific calculation method of the target loss function value is shown in the following formula (9):

[0168] Loss=w1*loss_proposal+w2*loss_desc_depth+w3*loss_depth+

[0169] +w5*loss_vis_pc(9)

[0170] Among them, w5 is the weight of the point cloud visualization loss value.

[0171] In summary, in this embodiment, a point cloud branch cutout is generated based on the second three-dimensional detection frame and the sample roadside point cloud data; a point cloud visualization loss value is calculated based on the point cloud branch cutout and the second depth map; and a target loss function value is calculated based on the target detection loss value, the depth description loss value, the depth estimation loss value, and the point cloud visualization loss value. Thus, the point cloud branch cutout is introduced to present a target object image in a small range of viewing angles, and the target loss function value is accurately calculated.

[0172] On the basis of the above embodiment, in another embodiment of the present application, the specific calculation method of the target loss function value can also be shown in the following formula (10):

[0173] Loss=w1*loss_proposal+w2*loss_desc_depth+w3*loss_depth+

[0174] w4*loss_vis_cam+w5*loss_vis_pc(10)

[0175] Among them, w1+w2+w3+w4+w5=1.

[0176] Based on the above embodiments, the embodiments of the present application also provide a target matching method.

[0177] Figure 8 FIG. 1 is a flowchart of a target matching method. Figure 8 As shown, the method includes:

[0178] S801. Acquire a target roadside image and corresponding target roadside point cloud data.

[0179] For example, multiple sets of target roadside images and target roadside point cloud data may be acquired and processed synchronously.

[0180] S802: Using the visual branch network and the point cloud branch network in the cross-modal matching model, the target roadside image and the target roadside point cloud data are processed respectively to obtain visual detection results and point cloud detection results.

[0181] The visual detection result includes: multiple first detection boxes, and the point cloud detection result includes: multiple second detection boxes; wherein the cross-modal matching model is a model trained by the method of any of the above embodiments.

[0182] S803: Perform target matching on the multiple first detection frames and the multiple second detection frames.

[0183] The similarities between the plurality of first detection frames and the plurality of second detection frames are calculated, and the second detection frame with the highest similarity to the first detection frame is used as the matching result of the first detection frame, thereby determining the object matching between the target roadside image and the target roadside point cloud data.

[0184] Furthermore, the visual detection result in the embodiment of the present application also includes: a plurality of first top-view depth description information, and the point cloud detection result includes: a plurality of second top-view depth description information.

[0185] It is also possible to perform target matching on multiple first bird's-eye view depth description information and multiple second bird's-eye view depth description information. Specifically, the similarity between multiple first bird's-eye view depth description information and multiple second bird's-eye view depth description information is calculated, and the second bird's-eye view depth description information with the highest similarity to the first bird's-eye view depth description information is used as the matching result of the first bird's-eye view depth description information. Then, the object matching between the target roadside image and the target roadside point cloud data is determined.

[0186] Furthermore, the similarity between the multiple first detection frames and the multiple second detection frames is calculated, the similarity between the multiple first overhead depth description information and the multiple second overhead depth description information is calculated, the similarities between the two are summed, and object matching is performed based on the summed similarity.

[0187] Thus, accurate object matching is achieved through the cross-modal matching model.

[0188] In summary, in this embodiment, the target roadside image and the corresponding target roadside point cloud data are obtained; the visual branch network and the point cloud branch network in the cross-modal matching model are used to process the target roadside image and the target roadside point cloud data respectively to obtain visual detection results and point cloud detection results, the visual detection results include: multiple first detection frames, and the point cloud detection results include: multiple second detection frames; wherein the cross-modal matching model is a model trained by the method of any one of the above embodiments; target matching is performed on multiple first detection frames and multiple second detection frames. Thus, accurate object matching is achieved through the cross-modal matching model.

[0189] Based on the above embodiments, the embodiments of the present application also provide the structure of a cross-modal matching model. Fig. 9 A structural diagram of the cross-modal matching model provided in an embodiment of the present application.

[0190] like Fig. 9As shown, the cross-modal matching model includes: a visual branch network and a point cloud branch network. The visual branch network includes: a visual encoder, a visual feature extractor, and a visual multi-head output. The input of the visual encoder is a roadside image, and the output of the visual multi-head output includes: a first three-dimensional detection frame, a first overhead depth description information, and an overhead image description information. The point cloud branch network includes: a point cloud encoder, a point cloud feature extractor, and a point cloud multi-head output. The input of the point cloud encoder is the roadside point cloud data, and the output of the point cloud multi-head output includes: a second three-dimensional detection frame and a second overhead depth description information.

[0191] The above structure can already complete the object matching task. On this basis, in order to complete the model training, the cross-modal matching model also includes: a deep vision decoder and an image decoder for training comparison. By visualizing the internal structure flow of the model, descriptor supervision within the modality and descriptor supervision between cross-modal (vision, point cloud) modalities are achieved.

[0192] The following describes the training, matching apparatus, device, and storage medium for executing a cross-modal matching model provided by the present application. The specific implementation process and technical effects are described above and will not be repeated below.

[0193] Fig.10 A schematic diagram of a training device for a cross-modal matching model provided in an embodiment of the present application, such as Fig.10 As shown, the cross-modal matching model includes: a visual branch network and a point cloud branch network, and the device includes:

[0194] The first acquisition module 1001 is used to acquire multiple groups of sample cross-modal sample data, each group of cross-modal sample data includes: a sample roadside image and corresponding sample roadside point cloud data.

[0195] The first detection module 1002 is used to use a preset visual branch network to detect the sample roadside images in each group of cross-modal sample data, and obtain the sample visual detection results corresponding to each group of cross-modal sample data. The sample visual detection results include: a first three-dimensional detection frame of the target object.

[0196] The second detection module 1003 is used to detect the sample roadside point cloud data in each group of cross-modal sample data using a preset point cloud branch network to obtain the sample point cloud detection results corresponding to each group of cross-modal sample data, and the sample point cloud detection results include: a second three-dimensional detection frame of the target object.

[0197] The first calculation module 1004 is used to calculate the target detection loss value according to the first three-dimensional detection frame, the first three-dimensional annotation frame of the target object in the sample roadside image, the second three-dimensional detection frame and the second three-dimensional annotation frame of the target object in the corresponding sample roadside point cloud data.

[0198] The second calculation module 1005 is used to calculate the target loss function value according to the target detection loss value.

[0199] The adjustment module 1006 is used to adjust the parameters of the preset visual branch network and the preset point cloud branch network to obtain a cross-modal matching model.

[0200] Furthermore, the first calculation module 1004 is specifically used to calculate a first detection loss value based on the first three-dimensional detection frame and the first three-dimensional annotation frame of the target object in the sample roadside image; calculate a second detection loss value based on the second three-dimensional detection frame and the corresponding second three-dimensional annotation frame of the target object in the sample roadside point cloud data; and calculate a target detection loss value based on the first detection loss value and the second detection loss value.

[0201] Furthermore, the first calculation module 1004 is specifically used for visual detection results also including: first overhead depth description information; the point cloud detection results also include: second overhead depth description information; according to the first overhead depth description information and the second overhead depth description information, the depth description loss value is calculated; according to the target detection loss value and the depth description loss value, the target loss function value is calculated.

[0202] Furthermore, the first calculation module 1004, which is specifically used for the cross-modal matching model, also includes: a depth vision decoder; using a depth vision decoder to generate a first depth map and a second depth map based on the first top-view depth description information and the second top-view depth description information; calculating a depth estimation loss value based on the first depth map and the second depth map; calculating a target loss function value based on the target detection loss value, the depth description loss and the depth estimation loss value.

[0203] Furthermore, the first calculation module 1004, which is specifically used for the cross-modal matching model, also includes: an image decoder; the visual detection result also includes: a bird's-eye view image description information; based on the bird's-eye view image description information, using an image decoder to generate a predicted color image corresponding to the sample roadside image; based on the first three-dimensional detection frame and the sample roadside image, generating a first visual branch cutout; based on the first three-dimensional detection frame and the predicted color image, generating a second visual branch cutout; based on the first visual branch cutout and the second visual branch cutout, calculating a visual visualization loss value; based on the target detection loss value, the depth description loss, the depth estimation loss value and the visual visualization loss value, calculating the target loss function value.

[0204] Furthermore, the first calculation module 1004 is specifically used to generate a first visual branch cutout based on the first three-dimensional detection frame, the first depth map and the sample roadside image; and generate a second visual branch cutout based on the first three-dimensional detection frame, the first depth map and the predicted color image.

[0205] Furthermore, the first calculation module 1004 is specifically used to generate a point cloud branch cutout based on the second three-dimensional detection frame and the sample roadside point cloud data; calculate the point cloud visualization loss value based on the point cloud branch cutout and the second depth map; calculate the target loss function value based on the target detection loss value, depth description loss value, depth estimation loss value and point cloud visualization loss value.

[0206] Fig.11 A schematic diagram of a target matching device provided in an embodiment of the present application is shown in FIG. Fig.11 As shown, the device comprises:

[0207] The second acquisition module 1101 is used to acquire a target roadside image and corresponding target roadside point cloud data.

[0208] The third detection module 1102 is used to use the visual branch network and the point cloud branch network in the cross-modal matching model to process the target roadside image and the target roadside point cloud data respectively to obtain visual detection results and point cloud detection results. The visual detection results include: multiple first detection frames, and the point cloud detection results include: multiple second detection frames; wherein the cross-modal matching model is a model trained by the method of any of the above embodiments.

[0209] The matching module 1103 performs target matching on the multiple first detection frames and the multiple second detection frames.

[0210] Fig.12 A schematic diagram of an electronic device provided in an embodiment of the present application, which electronic device may be a device with computing and processing functions.

[0211] The electronic device includes: a processor 1201 and a storage medium 1202. The processor 1201 and the storage medium 1202 are connected via a bus.

[0212] The storage medium 1202 is used to store programs, and the processor 1201 calls the programs stored in the storage medium 1202 to execute the above method embodiment. The specific implementation method and technical effect are similar and will not be repeated here.

[0213] Optionally, the present invention also provides a storage medium, including a program, which is used to execute the above-mentioned method embodiment when executed by a processor. In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0214] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0215] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0216] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to perform some steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (English: Read-Only Memory, abbreviated: ROM), random access memory (English: Random Access Memory, abbreviated: RAM), disk or optical disk and other media that can store program codes.

Claims

1. A method for training a cross-modal matching model, characterized in that: The cross-modal matching model includes: a visual branch network and a point cloud branch network, and the method includes: Acquire multiple groups of sample cross-modal sample data, each group of cross-modal sample data includes: a sample roadside image and corresponding sample roadside point cloud data; Using a preset visual branch network, the sample roadside image in each group of cross-modal sample data is detected to obtain a sample visual detection result corresponding to each group of cross-modal sample data, wherein the sample visual detection result includes: a first three-dimensional detection frame of the target object; Using a preset point cloud branch network, the sample roadside point cloud data in each group of cross-modal sample data is detected to obtain a sample point cloud detection result corresponding to each group of cross-modal sample data, wherein the sample point cloud detection result includes: a second three-dimensional detection frame of the target object; Calculating a target detection loss value according to the first three-dimensional detection frame, the first three-dimensional annotation frame of the target object in the sample roadside image, the second three-dimensional detection frame, and the corresponding second three-dimensional annotation frame of the target object in the sample roadside point cloud data; Calculating a target loss function value according to the target detection loss value; The preset visual branch network and the preset point cloud branch network are adjusted to obtain the cross-modal matching model.

2. The method according to claim 1, characterized in that The calculating the target detection loss value according to the first three-dimensional detection frame, the first three-dimensional annotation frame of the target object in the sample roadside image, the second three-dimensional detection frame, and the corresponding second three-dimensional annotation frame of the target object in the sample roadside point cloud data includes: Calculating a first detection loss value according to the first three-dimensional detection frame and a first three-dimensional annotation frame of the target object in the sample roadside image; Calculating a second detection loss value according to the second three-dimensional detection frame and the corresponding second three-dimensional annotation frame of the target object in the sample roadside point cloud data; The target detection loss value is calculated according to the first detection loss value and the second detection loss value.

3. The method according to claim 1, characterized in that The visual detection result also includes: first top-down depth description information; the point cloud detection result also includes: second top-down depth description information; Before calculating the target loss function value according to the target detection loss value, the method further includes: Calculating a depth description loss value according to the first top-view depth description information and the second top-view depth description information; The calculating the target loss function value according to the target detection loss value includes: The target loss function value is calculated according to the target detection loss value and the depth description loss value.

4. The method according to claim 3, characterized in that The cross-modal matching model further includes: a deep visual decoder; Before calculating the target loss function value according to the target detection loss value and the depth description loss value, the method further includes: Generate a first depth map and a second depth map using the depth vision decoder according to the first top-view depth description information and the second top-view depth description information; Calculating a depth estimation loss value according to the first depth map and the second depth map; The calculating the target loss function value according to the target detection loss value and the depth description loss value includes: The target loss function value is calculated according to the target detection loss value, the depth description loss and the depth estimation loss value.

5. The method according to claim 4, characterized in that The cross-modal matching model further includes: an image decoder; the visual detection result further includes: overhead image description information; before calculating the target loss function value according to the target detection loss value, the depth description loss and the depth estimation loss value, the method further includes: According to the overhead image description information, using the image decoder, generating a predicted color image corresponding to the sample roadside image; Generate a first visual branch cutout according to the first three-dimensional detection frame and the sample roadside image; Generating a second visual branch cutout according to the first three-dimensional detection frame and the predicted color image; Calculating a visual visualization loss value according to the first visual branch cutout and the second visual branch cutout; The calculating the target loss function value according to the target detection loss value, the depth description loss and the depth estimation loss value includes: The target loss function value is calculated according to the target detection loss value, the depth description loss, the depth estimation loss value and the visual visualization loss value.

6. The method according to claim 5, characterized in that The step of generating a first visual branch cutout according to the first three-dimensional detection frame and the sample roadside image includes: Generate the first visual branch cutout according to the first three-dimensional detection frame, the first depth map, and the sample roadside image; The step of generating a second visual branch cutout according to the first three-dimensional detection frame and the predicted color image includes: Generate the second visual branch cutout according to the first three-dimensional detection box, the first depth map and the predicted color image.

7. The method according to claim 4, characterized in that Before calculating the target loss function value according to the target detection loss value, the depth description loss and the depth estimation loss value, the method further includes: Generate a point cloud branch cutout according to the second three-dimensional detection frame and the sample roadside point cloud data; Calculating a point cloud visualization loss value according to the point cloud branch cutout and the second depth map; The calculating the target loss function value according to the target detection loss value, the depth description loss and the depth estimation loss value includes: The target loss function value is calculated according to the target detection loss value, the depth description loss, the depth estimation loss value and the point cloud visualization loss value.

8. A target matching method, characterized in that: The method comprises: Obtain target roadside image and corresponding target roadside point cloud data; The visual branch network and the point cloud branch network in the cross-modal matching model are used to process the target roadside image and the target roadside point cloud data respectively to obtain a visual detection result and a point cloud detection result, wherein the visual detection result includes: a plurality of first detection frames, and the point cloud detection result includes: a plurality of second detection frames; wherein the cross-modal matching model is a model trained by the method described in any one of claims 1 to 7 above; Object matching is performed on the multiple first detection frames and the multiple second detection frames.

9. An electronic device, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the training method for a cross-modal matching model as described in any one of claims 1 to 7, or the steps of the target matching method as described in claim 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the training method for a cross-modal matching model as described in any one of claims 1 to 7, or the steps of the target matching method as described in claim 8.

Images