Image mapping method, device, storage medium and electronic device

Abstract

The embodiment of the present invention provides an image mapping method, device, storage medium and electronic device, the method comprising: obtaining an initial pixel image and an initial radar image of a target scene obtained after synchronously shooting the target scene; determining the mapping coordinates of a target object included in the initial pixel image in the initial radar image, and determining the mapping coordinates as first coordinate information; establishing an initial association sequence between the pixel image and the radar image based on the first coordinate information; disassembling the initial association sequence to obtain a plurality of association subsequences; calculating loss values ​​for the association subsequences to obtain a target association sequence; and fusing the initial radar image with the initial pixel image according to the target association sequence. The present invention solves the problem of low image recognition accuracy in the related art, thereby achieving the effect of improving image recognition accuracy.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of communications, and in particular, to an image mapping method, device, storage medium and electronic device. Background Art

[0002] In intelligent monitoring, unambiguous and complete data is of paramount importance. With the development of intelligent monitoring equipment, the accuracy of monitoring objects in the scene is getting higher and higher. However, the monitoring accuracy of current intelligent monitoring equipment still has room for improvement.

[0003] For example, when monitoring a park, if the target recognition of pedestrians is incorrect, it will lead to errors in the monitoring of pedestrian trajectories, thus affecting subsequent early warnings and processing, such as sound warnings or pedestrian face capture.

[0004] Existing object recognition relies on high-precision sensors or high-precision calibration algorithms. The aforementioned recognition methods have high requirements for the use environment or require high technical support, and therefore cannot be applied to ordinary environments. If high-precision sensors or high-precision calibration algorithms are not used, the problem of low image recognition accuracy will arise.

[0005] However, there is no effective solution to the above problems at present. Summary of the invention

[0006] Embodiments of the present invention provide an image mapping method, device, storage medium and electronic device to at least solve the problem of low image recognition accuracy in the related art.

[0007] According to an embodiment of the present invention, there is provided an image mapping method, comprising:

[0008] Acquire an initial pixel image and an initial radar image of the target scene obtained after synchronously shooting the target scene;

[0009] Determining mapping coordinates of the target object included in the initial pixel image in the initial radar image, and determining the mapping coordinates as first coordinate information;

[0010] Based on the first coordinate information, establishing an initial association sequence between the initial pixel image and the initial radar image, wherein the initial association sequence is used to indicate a mapping relationship between the initial pixel image and the initial radar image;

[0011] Degrouping the initial associated sequence to obtain multiple associated subsequences;

[0012] Calculating the loss value of the associated subsequence to obtain a target associated sequence;

[0013] The initial radar image is fused with the initial pixel image according to the target association sequence.

[0014] In an exemplary embodiment, establishing an initial association sequence between the pixel image and the radar image based on the first coordinate information includes:

[0015] Determining second coordinate information of the target object included in the initial radar image;

[0016] Determine a distance difference between the first coordinate information and the second coordinate information;

[0017] Based on the distance difference, the initial correlation sequence between the initial pixel image and the initial radar image is determined.

[0018] In an exemplary embodiment, calculating the loss value of the associated subsequence to obtain a target associated sequence includes:

[0019] Calculating the loss value of the associated subsequence to obtain a target associated subsequence with the minimum loss value;

[0020] The target-associated subsequence is determined as the target-associated sequence.

[0021] In an exemplary embodiment, calculating the loss value of the associated subsequence to obtain a target associated subsequence with the minimum loss value includes:

[0022] determining a penalty value according to a degree of overlap between the first area information and the second area information;

[0023] Based on the penalty value, calculating the loss value of the associated subsequence to obtain the loss value of the associated subsequence;

[0024] Comparing the loss values ​​to obtain a minimum loss value;

[0025] The associated subsequence corresponding to the minimum loss value is determined as the target associated subsequence.

[0026] In an exemplary embodiment, disassembling the initial association sequence to obtain a plurality of association subsequences includes:

[0027] Decomposing the initial associated sequence according to a preset rule to obtain a plurality of subsequences;

[0028] A plurality of the subsequences are combined to obtain a plurality of the associated subsequences.

[0029] According to another embodiment of the present invention, there is provided an image mapping device, comprising:

[0030] An image acquisition module is used to obtain an initial pixel image and an initial radar image of the target scene after synchronously shooting the target scene;

[0031] A first coordinate module, configured to determine mapping coordinates of the target object included in the initial pixel image in the initial radar image, and determine the mapping coordinates as first coordinate information;

[0032] an initial sequence module, used to establish an initial association sequence between the pixel image and the radar image based on the first coordinate information, wherein the initial association sequence is used to indicate a mapping relationship between the pixel image and the radar image;

[0033] A sequence degrouping module, used for degrouping the initial associated sequence to obtain multiple associated subsequences;

[0034] A loss calculation module, used for calculating the loss value of the associated subsequence to obtain a target associated sequence;

[0035] A fusion module is used to fuse the initial radar image with the initial pixel image according to the target association sequence.

[0036] In an exemplary embodiment, the initial sequence module includes:

[0037] A second coordinate unit, used to determine second coordinate information of the target object included in the initial radar image;

[0038] a difference calculation unit, used to determine a distance difference between the first coordinate information and the second coordinate information;

[0039] A sequence determination unit is used to determine the initial association sequence between the initial pixel image and the initial radar image based on the distance difference.

[0040] In an exemplary embodiment, the loss calculation module includes:

[0041] A loss calculation unit, used for calculating the loss value of the associated subsequence to obtain a target associated subsequence with the minimum loss value;

[0042] The target sequence determining unit is used to determine the target associated subsequence as the target associated sequence.

[0043] According to yet another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any one of the above method embodiments when run.

[0044] According to yet another embodiment of the present invention, there is provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0045] Through the present invention, since the target object in the pixel image is mapped to the radar image and the pixel image and the radar image are fused, the target object in the pixel image can be accurately identified, thereby improving the recognition accuracy of the pixel image. Therefore, the problem of low image recognition accuracy in the related art can be solved, and the effect of improving image recognition accuracy can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a hardware structure block diagram of a mobile terminal of an image mapping method according to an embodiment of the present invention;

[0047] Figure 2 is a flow chart of an image mapping method according to an embodiment of the present invention;

[0048] Figure 3 is an exemplary diagram showing a mapping process according to an embodiment of the present invention;

[0049] Figure 4 is a structural block diagram of an image mapping device according to an embodiment of the present invention;

[0050] Figure 5 is an initial radar image and an initial pixel image acquired according to a specific embodiment of the embodiment of the present invention;

[0051] Figure 6 is a flow chart according to a specific embodiment of the present invention. DETAILED DESCRIPTION

[0052] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings and in combination with the embodiments.

[0053] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0054] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 FIG. 1 is a hardware structure diagram of a mobile terminal of an image mapping method according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0055] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as a computer program corresponding to an image mapping method in an embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0056] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0057] In this embodiment, an image mapping method is provided. Figure 2 is a flow chart of an image mapping method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0058] Step S202, acquiring an initial pixel image and an initial radar image of the target scene obtained after synchronously photographing the target scene;

[0059] In this embodiment, the initial pixel image and the initial radar image of the target scene obtained after synchronous shooting are obtained to ensure that the timestamps of the initial pixel image and the initial radar image are consistent, thereby reducing the interference of images with inconsistent timestamps on image recognition, thereby improving image recognition efficiency and accuracy.

[0060] The initial pixel image may be (but not limited to) a picture or a video, and the number of pictures may be one frame, thereby reducing the amount of calculation, or multiple frames, thereby improving the accuracy of image calibration; the initial pixel image may be (but not limited to) an infrared image, or a visible light image, or a fusion of the two; the acquisition of the initial pixel image may be (but not limited to) acquired by collecting the target area through a visible light camera or a visible light sensor, or by collecting the target area through an infrared camera or an infrared sensor; the initial radar image may be (but not limited to) acquired by collecting the target scene through a radar sensor; synchronous shooting may be (but not limited to) achieved by synchronous processing of the image acquisition device, or by synchronizing the acquired initial pixel image and the initial radar image according to the timestamp, or by processing in other ways; the target scene may be an ideal scene that meets the preset conditions, wherein the ideal environment refers to an environment in which there are very sparse target objects, the measurement quality of the target objects is good, and the radar video target matching is unambiguous.

[0061] Step S204, determining the mapping coordinates of the target object included in the initial pixel image in the initial radar image, and determining the mapping coordinates as first coordinate information;

[0062] In this embodiment, the target object included in the initial pixel image is mapped into the initial radar image in order to determine the relationship between the target object in the initial radar image and the target object included in the initial pixel image, thereby facilitating tracking of the target object.

[0063] Among them, the determination process of the mapping coordinates can be (but not limited to) mapping the pixel points (U, V) in the initial pixel image to the 2D position coordinates (X, Y) in the initial radar image. The determination method can be implemented by a camera internal and external parameter calibration method based on the internal and external parameters of the image acquisition device, or it can be implemented by a four-point calibration method; the determination process can be implemented by a neural network model, or it can be implemented by a combination of a preset algorithm and a neural network; the execution of the determination process can be (but not limited to) implemented by built-in CPU, FPGA and other logical operation modules, or it can be implemented by an external computer or cloud processing terminal. When the scene device is implemented by an external device, its data transmission can be implemented through Ethernet, or it can be implemented through a wireless transmission network such as 3G / 4G / 5G, or it can be implemented by other methods; the first coordinate information can be (but not limited to) the coordinate information, area size, identification ID of the target object in the initial pixel image mapped to the initial radar image.

[0064] Step S206, establishing an initial association sequence between the pixel image and the radar image based on the first coordinate information, wherein the initial association sequence is used to indicate a mapping relationship between the initial pixel image and the initial radar image;

[0065] In this embodiment, the initial association sequence is established to determine the mapping relationship between the initial pixel image and the initial radar image.

[0066] The established association sequence may be as shown in Table 1:

[0067] Table 1

[0068] Radar / Video ID 1 2 3 A Dis11 Dis12 Dis13 B Dis21 Dis22 Dis23 C Dis31 Dis32 Dis33

[0069] In the table, the rows represent the target objects in the initial radar image, and the columns represent the target objects in the initial pixel image; Disij represents the distance between the i-th radar target and the j-th video target in the rectangular coordinate system, and satisfies Disij>0.

[0070] Furthermore, the correlation matrix shown in Table 1 can be further simplified as shown in Table 2:

[0071] Table 2

[0072] Dis11 Dis12 Dis13 Dis21 Dis22 Dis23 Dis31 Dis32 Dis33

[0073] Step S208, disassembling the initial associated sequence to obtain multiple associated subsequences;

[0074] In this embodiment, the initial association sequence is disassembled to obtain mapping modes of different target objects, thereby obtaining an optimal mapping mode.

[0075] The disassembly of the initial association sequence may include two steps of disassembly and reassembly, or may include only one step of disassembly; the disassembly of the initial association sequence may be one of the following or a combination thereof:

[0076] 1) The number of non-zero elements in any row is at most 1 (indicating that a video target is associated with at most one radar target);

[0077] 2) The number of non-zero elements in any column is at most 1 (indicating that a radar target is associated with at most one video target).

[0078] Among them, based on the above rules, the partial decomposition results of the association sequence corresponding to Table 2 are as follows:

[0079]

[0080] In the first group, only the diagonal elements are non-zero, and the association list expressed is: A-1, B-2, C-3.

[0081] The association list expressed by the second group is: A-2, B-1, C-3.

[0082] The association list expressed by the fourth group is: A-0, B-2, C-3, 0-1.

[0083] The association list expressed by the 6th group is: A-0, B-1, C-0, 0-2, 0-3.

[0084] It should be noted that the above sequence decomposition results are only part of all decomposition results, and there are many other correlation sub-matrices that are not presented. And each correlation sub-matrix uniquely corresponds to a correlation list, and all correlation sub-matrices can describe all correlation lists.

[0085] Step S2010, calculating the loss value of the associated subsequence to obtain a target associated sequence;

[0086] In this embodiment, the loss value is calculated in order to obtain a mapping method with the least loss of elements such as pixels during the mapping process.

[0087] Among them, the loss value can be calculated by the following formula:

[0088] loss k =∑Dis(Vid-Rid)

[0089] In the formula, loss k is the loss value of the kth associated submatrix.

[0090] For example, for group 1, the loss is: Dis11+Dis22+Dis33.

[0091] For the 6th group, the loss value is: Dis21+4*DisNon.

[0092] Where DisNon is the penalty value, and the penalty value is used to set a default distance value for the target matching result that is not associated. This distance value is usually relatively large. For example, in the fourth group of associated submatrices in the above sequence, A-0 means that the target object A in the pixel image is not associated with any target object in any radar image, and 0-1 means that the target object 1 in the radar image is not associated with any target object in the pixel image, that is:

[0093] Dis(A-0) = DisNon;

[0094] Dis(0-1)=DisNon.

[0095] Step S2012: fusing the initial radar image with the initial pixel image according to the target association sequence.

[0096] In this embodiment, the initial radar image and the initial pixel image are fused so that the user can understand the contents of the initial radar image and the pixel image at the same time.

[0097] The fusion operation may (but is not limited to) be displaying the initial radar image and the initial pixel image side by side in the same visualization interface in the same size, or displaying them side by side in the same visualization interface in a large and a small manner, or displaying the larger image in the center of the visualization interface and the smaller image at the edge of the visualization interface (for example, the lower right corner) in a large and a small manner; the fusion operation may also be capturing a target object in one of the images and pasting it to a corresponding position in the other image, or other fusion methods.

[0098] Through the above steps, since the target object in the pixel image is mapped to the radar image and the pixel image and the radar image are fused, the target object in the pixel image can be accurately identified, thereby improving the recognition accuracy of the pixel image.

[0099] The execution subject of the above steps may be a base station, a terminal, etc., but is not limited thereto.

[0100] In an optional embodiment, determining the mapping coordinates of the target object included in the initial pixel image in the initial radar image, and determining the mapping coordinates as the first coordinate information includes:

[0101] Step S2042, performing identification processing on the initial pixel image to obtain the center coordinate information of the target object in the initial pixel image;

[0102] Step S2044: Determine the mapping coordinates of the target object included in the initial pixel image in the initial radar image according to the center coordinate information.

[0103] In this embodiment, the purpose of marking the initial pixel image is to determine the position of the target object in the initial pixel image, and then mapping the initial pixel image is performed to reduce errors generated in the mapping process.

[0104] The identification processing of the initial pixel image may be to identify the target object in the initial pixel image with an identification frame, or to identify the outline of the target object by a four-point identification method, and then frame the target object with an identification frame of a specific shape. It should be noted that the identification frame may be square, circular, or other shapes, or a curved frame for identifying the edge points of the target object located at the edge; the size of the identification frame may be (but not limited to) determined according to the difference between the maximum coordinate and the minimum coordinate of the four-point identification, or may be a fixed size; in the case of using the four-point identification method for identification, the position of the center point of the target object may be determined, thereby obtaining the center coordinate information of the target object in the initial pixel image, and the center coordinate information includes (but not limited to) the coordinates of the center point of the target object, the identification ID and other information; the mapping coordinates include the coordinate information of the center point of the target object mapped to the initial radar image, wherein the mapping method may be implemented by a camera internal and external parameter calibration method and / or a four-point calibration method.

[0105] For example, Figure 3 As shown, the center point of the target object is obtained based on the target box, such as Figure 3 In the middle left figure, the center point of the target object is extracted based on the four identification points; then according to the preset calibration function, the corresponding radar coordinates are obtained based on the center point of each target; Figure 3 Right Figure 3 A rectangular solid box. At this time, all the marks in the right image are points corresponding to the center point of the pixel image, rather than areas.

[0106] In an optional embodiment, establishing an initial association sequence between the pixel image and the radar image based on the first coordinate information includes:

[0107] Step S2062, determining second coordinate information of the target object included in the initial radar image;

[0108] Step S2064, determining the distance difference between the first coordinate information and the second coordinate information;

[0109] Step S2066: determining an initial correlation sequence between the initial pixel image and the initial radar image based on the distance difference.

[0110] In this embodiment, the second coordinate information of the target object included in the initial radar image is determined in order to determine the position of the target object in the initial radar image, so as to facilitate determination of the coordinate difference between the target object in the initial radar image and the target object in the initial pixel image, that is, to determine the distance difference between the first coordinate information and the second coordinate information, thereby judging whether the mapping method is accurate.

[0111] Among them, the second coordinate information includes (but is not limited to) the coordinates of the target object included in the initial radar image, the identification ID of the target object and other information; the determination of the second coordinate information can be (but is not limited to) obtained by scanning the acquired radar image through a radar image acquisition device, or it can be obtained by directly recording the radar coordinates of the target object when the radar image of the target scene is acquired, or it can be obtained by other methods; the calculation of the distance difference is to calculate the difference between the coordinates of the center point of the target object in the initial pixel image and the coordinates of the target object in the initial radar image, so as to determine whether the mapping coordinates of the target object in the initial pixel image coincide with the radar coordinates of the target object in the initial pixel image, so as to determine whether the mapping is accurate; the calculation of the distance difference can be (but is not limited to) implemented by built-in logic operation modules such as CPU and FPGA, and can also be implemented by an external computer or cloud processing end. In the case of implementing the scene device through an external device, its data transmission can be implemented through Ethernet, or through a wireless transmission network such as 3G / 4G / 5G, or it can be implemented by other methods.

[0112] It should be noted that the calculated distance difference is the value of Disij in the aforementioned Table 1 and Table 2; wherein, for the convenience of device identification, the value of Disij greater than the preset threshold can be set to 1, and the value of Disij less than the preset threshold can be set to 0, or the opposite setting method can be used, that is, the value of Disij greater than the preset threshold is set to 0, and the value of Disij less than the preset threshold is set to 1, or other settings can be used; and when the distance is 0, it is defined as:

[0113] Dis(Vid-Rid)=DisNon, where DisNon is the penalty value;

[0114] It should be noted that the penalty value is used to set a default distance value for the target matching result without association, and the distance value is usually relatively large. For example, in the fourth group of association submatrices in the above sequence, A-0 means that the target object A in the pixel image is not associated with any target object in any radar image, and 0-1 means that the target object 1 in the radar image is not associated with any target object in the pixel image, that is:

[0115] Dis(A-0) = DisNon;

[0116] Dis(0-1)=DisNon.

[0117] In an optional embodiment, calculating the loss value of the associated subsequence to obtain the target associated sequence includes:

[0118] Step S20102, calculating the loss value of the associated subsequence to obtain a target associated subsequence with the minimum loss value;

[0119] Step S20104: determine the target associated subsequence as the target associated sequence.

[0120] In this embodiment, determining the target association subsequence with the minimum loss value is to determine the optimal association method between the initial radar image and the initial pixel image.

[0121] Among them, the loss value can be calculated by the following formula:

[0122] loss k =∑Dis(Vid-Rid)

[0123] The correlation submatrix with the smallest loss value can be expressed as:

[0124] AssoMat=min{loss k}

[0125] For example, in the above sequence, for group 1, the loss is: Dis11+Dis22+Dis33.

[0126] For the 6th group, the loss value is: Dis21+4*DisNon.

[0127] In an optional embodiment, calculating the loss value of the associated subsequence to obtain a target associated subsequence with the minimum loss value includes:

[0128] Step S201022, determining a penalty value according to a distance difference between the first coordinate information and the second coordinate information;

[0129] Step S201024, based on the penalty value, calculating the loss value of the associated subsequence to obtain the loss value of the associated subsequence;

[0130] Step S201026, comparing the loss values ​​to obtain a minimum loss value;

[0131] Step S201028: determine the associated subsequence corresponding to the minimum loss value as the target associated subsequence.

[0132] In this embodiment, the loss value comparison method can be (but not limited to) comparing the obtained loss value with all the loss values ​​in turn and sorting them to obtain them, or it can be compared two by two, and the smaller value in the two-by-two comparison is compared with the smaller value obtained by another group of comparisons, and so on. It can also be obtained by other comparison methods.

[0133] In an optional embodiment, disassembling the initial association sequence to obtain a plurality of association subsequences includes:

[0134] Step S2082, disassembling the initial associated sequence according to a preset rule to obtain multiple subsequences;

[0135] Step S2084: combine multiple subsequences to obtain multiple associated subsequences.

[0136] In this embodiment, the initial association sequence is decomposed to determine the mapping results of target objects in different regions; and the multiple subsequences are combined to determine different mapping modes.

[0137] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0138] In this embodiment, an image mapping device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0139] Figure 4 is a structural block diagram of an image mapping device according to an embodiment of the present invention. Figure 4 As shown, the device comprises:

[0140] An image acquisition module 42 is used to obtain an initial pixel image and an initial radar image of the target scene after synchronously shooting the target scene;

[0141] A first coordinate module 44, configured to determine a mapping coordinate of a target object included in the initial pixel image in the initial radar image, and determine the mapping coordinate as first coordinate information;

[0142] An initial sequence module 46, used to establish an initial association sequence between the pixel image and the radar image based on the first coordinate information, wherein the initial association sequence is used to indicate a mapping relationship between the pixel image and the radar image;

[0143] A sequence degrouping module 48, configured to degroup the initial associated sequence to obtain a plurality of associated subsequences;

[0144] A loss calculation module 410 is used to calculate the loss value of the associated subsequence to obtain a target associated sequence;

[0145] The fusion module 412 fuses the initial radar image with the initial pixel image according to the target association sequence.

[0146] In an optional embodiment, the first coordinate module 44 includes:

[0147] The identification unit 442 performs identification processing on the initial pixel image to obtain the center coordinate information of the target object in the initial pixel image;

[0148] The mapping unit 444 determines the mapping coordinates of the target object included in the initial pixel image in the initial radar image according to the center coordinate information.

[0149] In an optional embodiment, the initial sequence module 46 includes:

[0150] A second coordinate unit 462, used to determine second coordinate information of the target object included in the initial radar image;

[0151] A difference calculation unit 464, used to determine a distance difference between the first coordinate information and the second coordinate information;

[0152] The sequence determination unit 466 is configured to determine an initial correlation sequence between the initial pixel image and the initial radar image based on the distance difference.

[0153] In an optional embodiment, the loss calculation module 410 includes:

[0154] A loss calculation unit 4102 is used to calculate the loss value of the associated subsequence to obtain a target associated subsequence with the minimum loss value;

[0155] The target sequence determining unit 4104 is configured to determine the target associated subsequence as a target associated sequence.

[0156] In an optional embodiment, the loss calculation unit 4102 includes:

[0157] A penalty value determination subunit 41022, configured to determine a penalty value according to a distance difference between the first coordinate information and the second coordinate information;

[0158] The loss value calculation subunit 41024 is used to calculate the loss value of the associated subsequence based on the penalty value to obtain the loss value of the associated subsequence;

[0159] A comparison subunit 41026, for comparing the loss values ​​to obtain a minimum loss value;

[0160] The sequence determination subunit 41028 is used to determine the associated subsequence corresponding to the minimum loss value as the target associated subsequence.

[0161] In an optional embodiment, the sequence disassembly module 48 includes:

[0162] A sequence disassembling unit 482, configured to disassemble the initial associated sequence according to a preset rule to obtain a plurality of subsequences;

[0163] The sequence combining unit 484 is used to combine multiple subsequences to obtain multiple associated subsequences.

[0164] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0165] The present invention is described below in conjunction with specific embodiments.

[0166] like Figure 5 As shown, video intelligent image (corresponding to Figure 5 Left image in the figure) and radar tracking image (corresponding to Figure 5 The right figure in the figure can accurately detect the number and position of pedestrians (except for the case where the target is outside the monitoring range of the sensor and is blocked). Among them, the video tracking algorithm can give the position of the target frame in real time. Therefore, the target sequence and the corresponding target frame sequence can be established:

[0167] Vtrg1, BoxPt1

[0168] Vtrg2, BoxPt2

[0169] Vtrg3, BoxPt3

[0170] …

[0171] Among them, Vtrg i Represents the video target sequence, BoxPt iIndicates the position of the video target frame, that is:

[0172]

[0173] In the formula, Represents the pixel coordinates of the upper left, upper right, lower left, and lower right points respectively.

[0174] Similarly, the radar can detect and track the number and position of pedestrians, and the following target sequence can be established:

[0175] Rtrg1, CartPt1

[0176] Rtrg2, CartPt2

[0177] Rtrg3, CartPt3

[0178] …

[0179] Where Rtrg i Represents the radar target sequence, CartPt i Indicates the radar target point position.

[0180] The video radar measurement data set is then processed by the matching algorithm. Based on the video radar measurement data set, the following mapping relationship can be constructed:

[0181] R i =F(Vtrg j ) (1)

[0182] Vtg i =G(Rtrg j ) (2)

[0183] F=G -1 (3)

[0184] G=F -1 (4)

[0185] Among them, F(*) maps the video target ID to the radar target ID one by one, and G(*) maps the radar target ID to the video target ID one by one.

[0186] Among them, F(*) and G(*) can be obtained as follows:

[0187] First, the radar 2D position (X, Y) corresponding to the video pixel point (U, V) is obtained by the camera internal and external parameter calibration method and / or the four-point calibration method, and then the association sequence shown in Table 1 and Table 2 is established, and the association sequence is disassembled according to the following rules:

[0188] 1) The number of non-zero elements in any row is at most 1 (indicating that a radar target is associated with at most one video target);

[0189] 2) The number of non-zero elements in any column is at most 1 (indicating that a video target is associated with at most one radar target);

[0190] Based on this, some feasible / possible disassembly result sequences are listed below:

[0191]

[0192] In the first group, only the diagonal elements are non-zero, and the association list expressed is: A-1, B-2, C-3.

[0193] The association list expressed by the second group is: A-2, B-1, C-3.

[0194] The association list expressed by the fourth group is: A-0, B-2, C-3, 0-1.

[0195] The association list expressed by the 6th group is: A-0, B-1, C-0, 0-2, 0-3.

[0196] It should be noted that there are many other correlation sub-matrices that are not presented. Each correlation sub-matrix uniquely corresponds to a correlation list and also corresponds to a possible correlation / matching relationship. All correlation sub-matrices can describe all matching relationships.

[0197] Then, a penalty value is set according to the above sequence, and the loss value of the sequence is calculated to obtain the association sequence with the minimum loss value, and the association sequence with the minimum loss value is used as the target association sequence, and then the images are fused according to the target association sequence.

[0198] The above steps are as follows Figure 6 As shown:

[0199] Step 1: Initialization. Set the penalty value, fusion target type, etc. (corresponding to Figure 6 Then, go to Step 1) and jump to Step 2.

[0200] Step 2: Camera internal and external parameters / four-point marking. The purpose of this step is to configure the parameters that the video radar calibration method depends on (corresponding to Figure 6 2) and jump to Step 3 and Step 7.

[0201] Step 3: Video target detection (corresponding to Figure 6 In step 3). In this embodiment, artificial intelligence methods such as deep learning are used to detect the number and position of targets, and each target is given a unique ID, and then jump to Step 4.

[0202] Step 4: Video target tracking (corresponding to Figure 6The purpose of this step is to have a stable ID for the same target and jump to Step 5.

[0203] Step 5: Extract the target center point (corresponding to Figure 6 Based on the position of the target frame in the video, extract the target center point and jump to Step 6.

[0204] Step 6: Get the virtual radar position (corresponding to Figure 6 Step 6 in the video). Based on the position of the target frame in the video, the target position in the corresponding radar coordinate system is calculated according to the video radar calibration function. Because the point position of the target in the radar coordinate system is calculated, it is recorded as the virtual radar target position, corresponding to Figure 3 Click the rectangle in the right picture and jump to Step 8.

[0205] Step 7: Radar sensor target (corresponding to Figure 6 Step 7). The radar sensor can obtain the number of targets and the corresponding target point positions, corresponding to Figure 3 Click the circle in the right picture and jump to Step 8.

[0206] Step 8: Time synchronization (corresponding to Figure 6 Based on the timestamp, paired radar and video data can be obtained and jump to Step 9.

[0207] Step 9: Create an association matrix (corresponding to Figure 6 9) and jump to Step 10.

[0208] Step 10: Decompose the association matrix into sub-association matrices (corresponding to Figure 6 Go to step 10) and jump to step 11.

[0209] Step 11: Get the associated list of sub-arrays (corresponding to Figure 6 For each sub-association matrix, obtain its corresponding association list and jump to Step 12.

[0210] Step 12: Calculate the loss value (corresponding to Figure 6 For each associated sub-matrix, calculate its loss value and jump to Step 13.

[0211] Step 13: Select the best association (corresponding to Figure 6 From all the associated sub-matrices and their corresponding loss values, select the associated sub-matrix with the smallest loss value and use it as the fusion result of the current frame. Jump to Step 14.

[0212] Step 14: End.

[0213] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0214] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0215] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0216] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0217] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0218] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0219] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An image mapping method, characterized in that: include: Acquire an initial pixel image and an initial radar image of the target scene obtained after synchronously shooting the target scene; Determining mapping coordinates of a target object included in the initial pixel image in the initial radar image, and determining the mapping coordinates as first coordinate information; Based on the first coordinate information, establishing an initial association sequence between the initial pixel image and the initial radar image, wherein the initial association sequence is used to indicate a mapping relationship between the initial pixel image and the initial radar image; Degrouping the initial associated sequence to obtain multiple associated subsequences; Calculating the loss value of the associated subsequence to obtain a target associated sequence; fusing the initial radar image with the initial pixel image according to the target association sequence; Among them, establishing an initial association sequence between the initial pixel image and the initial radar image based on the first coordinate information includes: determining second coordinate information of the target object included in the initial radar image; and determining the initial association sequence between the initial pixel image and the initial radar image based on the first coordinate information and the second coordinate information.

2. The method according to claim 1, characterized in that Determining the initial association sequence between the initial pixel image and the initial radar image based on the first coordinate information and the second coordinate information includes: Determine a distance difference between the first coordinate information and the second coordinate information; Based on the distance difference, the initial correlation sequence between the initial pixel image and the initial radar image is determined.

3. The method according to claim 2, characterized in that Calculating the loss value of the associated subsequence to obtain a target associated sequence includes: Calculating the loss value of the associated subsequence to obtain a target associated subsequence with the minimum loss value; The target-associated subsequence is determined as the target-associated sequence.

4. The method according to claim 3, characterized in that Calculating the loss value of the associated subsequence to obtain a target associated subsequence with the minimum loss value includes: Determine a penalty value according to a distance difference between the first coordinate information and the second coordinate information; Based on the penalty value, calculating the loss value of the associated subsequence to obtain the loss value of the associated subsequence; Comparing the loss values ​​to obtain a minimum loss value; The associated subsequence corresponding to the minimum loss value is determined as the target associated subsequence.

5. The method according to claim 1, characterized in that Decomposing the initial associated sequence to obtain multiple associated subsequences includes: Decomposing the initial associated sequence according to a preset rule to obtain a plurality of subsequences; A plurality of the subsequences are combined to obtain a plurality of the associated subsequences.

6. An image mapping device, characterized in that: include: An image acquisition module is used to obtain an initial pixel image and an initial radar image of the target scene after synchronously shooting the target scene; A first coordinate module, used to determine the mapping coordinates of the target object included in the initial pixel image in the initial radar image, and determine the mapping coordinates as first coordinate information; an initial sequence module, used to establish an initial association sequence between the initial pixel image and the initial radar image based on the first coordinate information, wherein the initial association sequence is used to indicate a mapping relationship between the initial pixel image and the initial radar image; A sequence degrouping module, used for degrouping the initial associated sequence to obtain multiple associated subsequences; A loss calculation module, used for calculating the loss value of the associated subsequence to obtain a target associated sequence; a fusion module, fusing the initial radar image with the initial pixel image according to the target association sequence; The initial sequence module is used to establish an initial association sequence between the initial pixel image and the initial radar image in the following manner: determining second coordinate information of the target object included in the initial radar image; and determining the initial association sequence between the initial pixel image and the initial radar image based on the first coordinate information and the second coordinate information.

7. The device according to claim 6, characterized in that The initial sequence module is used to determine the initial association sequence between the initial pixel image and the initial radar image based on the first coordinate information and the second coordinate information in the following manner: determine a distance difference between the first coordinate information and the second coordinate information; and determine the initial association sequence between the initial pixel image and the initial radar image based on the distance difference.

8. The device according to claim 6, characterized in that The loss calculation module includes: A loss calculation unit, used for calculating the loss value of the associated subsequence to obtain a target associated subsequence with the minimum loss value; The target sequence determining unit is used to determine the target associated subsequence as the target associated sequence.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 5 when executed.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 5.

Images