Image calibration method, device, storage medium and electronic device

Abstract

The embodiment of the present invention provides an image calibration method, device, storage medium and electronic device, wherein the method comprises: obtaining an initial pixel image and an initial radar image of a target scene; performing a first calibration process on the initial pixel image to obtain a target pixel image; performing a second calibration process on the initial radar image to obtain a target radar image; performing a fusion operation on the target radar image and the target pixel image to obtain a target data image, wherein the target data image comprises the target radar image and the target pixel image. Through the present invention, the problem of low image recognition accuracy and efficiency is solved, thereby achieving the effect of improving image recognition accuracy and efficiency.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of communications, and in particular, to an image calibration method, device, storage medium and electronic device. Background Art

[0002] In modern society, security is increasingly valued by the public, security products are emerging in an endless stream, security application areas are constantly expanding, and security technology is developing rapidly. However, due to special usage scenarios, security terminals still have problems with low recognition efficiency and poor recognition accuracy.

[0003] For example, traditional security terminal equipment is mainly visible light cameras. The advantage of visible light cameras is that they can obtain target category information, but it is not easy to obtain target operation information. For example, it is easy to identify the target type (human, not machine), but it is not easy to estimate the target's speed and spatial position. The recognition accuracy of targets at long distances, at night, and in rainy and foggy days is lower than that of targets at close range, that is, the time and space accuracy of targets in the monitoring area is inconsistent (even with significant differences).

[0004] Millimeter-wave radar actively emits electromagnetic waves and receives signals of the same frequency. It has a very high detection probability for moving objects or objects with a large RCS (radar cross-section area), but has a lower detection probability for stationary objects (the detection probability is not zero).

[0005] Currently, no effective solution has been proposed to address the problems of poor recognition accuracy and low recognition efficiency in related technologies. Summary of the invention

[0006] The embodiments of the present invention provide an image calibration method, device, storage medium and electronic device to at least solve the problems of poor recognition accuracy and low recognition efficiency in the related art.

[0007] According to an embodiment of the present invention, there is provided an image calibration method, comprising:

[0008] Acquire an initial pixel image and an initial radar image of a target scene, wherein the initial pixel image includes pixel information of a target object contained in the target scene, the initial radar image includes radar information of the target object, and the initial pixel image and the initial radar image are obtained by synchronously shooting the target scene;

[0009] Performing a first calibration process on the initial pixel image to obtain a target pixel image, wherein the target pixel image is a calibration image of the initial pixel image, and the target pixel image includes the initial pixel image and radar information of the target object;

[0010] Performing a second calibration process on the initial radar image to obtain a target radar image, wherein the target radar image is a calibrated image of the initial radar image, and the target radar image includes pixel information of the initial radar image and the target object;

[0011] A fusion operation is performed on the target radar image and the target pixel image to obtain a target data image, wherein the target data image includes the target radar image and the target pixel image.

[0012] In an exemplary embodiment, before acquiring the initial pixel image and the initial radar image of the target scene, the method includes: performing scene recognition on the target scene to determine whether the target scene satisfies a first condition;

[0013] The acquiring the initial pixel image and the initial radar image of the target scene includes: acquiring the initial pixel image and the initial radar image of the target scene when the target scene satisfies the first condition.

[0014] In an exemplary embodiment, performing a first calibration process on the initial pixel image to obtain a target pixel image includes:

[0015] The initial pixel image is subjected to a first calibration process using a first model to obtain a target pixel image, wherein the first model is trained through machine learning using a first data set, the first data set includes multiple groups of first data, and each group of the first data includes: a pixel image and a calibrated image of the pixel image.

[0016] In an exemplary embodiment, before performing the first calibration process on the initial pixel image, the method further includes:

[0017] Acquire a first initial data set, wherein the first initial data set includes multiple groups of first data, and each group of first data includes the pixel image and a calibration image of the pixel image;

[0018] Determining whether the first initial data set satisfies a second condition;

[0019] When the first initial data set satisfies a second condition, the first initial data set is determined as the first data set.

[0020] In an exemplary embodiment, determining whether the first initial data set satisfies the second condition includes:

[0021] The following operations are performed on each set of data included in the first initial data set to determine a set of data included in the first initial data set whose quality meets a predetermined quality threshold:

[0022] Extracting a first target area image contained in the pixel image;

[0023] Performing downsampling processing on the first target area image to obtain a first area image;

[0024] Binarizing the first region image to obtain a second region image;

[0025] Performing a binarization result detection on the second region image to obtain a binarization detection result;

[0026] determining whether the pixel image meets the quality threshold based on the binarization detection result, wherein if the binarization result is a first result, determining that the pixel image meets the quality threshold, and if the binarization result is a second result, determining that the pixel image does not meet the quality threshold;

[0027] Determine a first number of data groups included in the first initial data set whose quality satisfies the above-mentioned predetermined quality threshold;

[0028] Based on the first quantity, it is determined whether the first initial data set meets the second condition, wherein when the first quantity is greater than or equal to a quantity threshold, it is determined that the first initial data set meets the second condition, and when the first quantity is less than the quantity threshold, it is determined that the first initial data set does not meet the second condition.

[0029] In an exemplary embodiment, determining whether the first initial data set satisfies the second condition includes:

[0030] Acquire the number of data groups included in the first initial data set, and acquire the quality of each data group included in the first initial data set;

[0031] Whether the first initial data set satisfies the second condition is determined based on the quantity and the quality, wherein, when it is determined that the quantity satisfies the quantity condition and the quality satisfies the quality condition, it is determined whether the first initial data set satisfies the second condition, and when it is determined that the quantity does not satisfy the quantity condition and / or the quality does not satisfy the quality condition, it is determined that the first initial data set does not satisfy the second condition.

[0032] In an exemplary embodiment, after determining whether the first initial data set meets the second condition based on the quantity and the quality, the method further includes:

[0033] In the case where it is determined that the quantity does not satisfy the quantity condition, adding a data group included in the first initial data set;

[0034] In the case where it is determined that the quality does not satisfy the quality condition, all data groups included in the first data set are deleted, and the data groups included in the first data set are re-determined.

[0035] In an exemplary embodiment, performing a second calibration process on the initial radar image to obtain a target radar image includes:

[0036] The initial radar image is subjected to a second calibration process using a second model to obtain a target radar image, wherein the second model is trained by machine learning using a second data set, and the second data set includes multiple groups of second data, and each group of the second data includes: a radar image and a calibrated image of the radar image.

[0037] In an exemplary embodiment, before performing the second calibration process on the initial radar image, the method further includes:

[0038] Acquire a second initial data set, wherein the second initial data set includes multiple groups of second data, and each group of second data includes the radar image and a calibration image of the radar image;

[0039] Determining whether the second initial data set satisfies a second condition;

[0040] When the second initial data set satisfies a second condition, the second initial data set is determined as the second data set.

[0041] In an exemplary embodiment, determining whether the second initial data set satisfies the second condition includes:

[0042] The following operations are performed for each set of data included in the second initial data set to determine a set of data included in the second initial data set whose quality meets a predetermined quality threshold:

[0043] Extracting a second target area image contained in the radar image;

[0044] Downsampling the second target area image to obtain a third area image;

[0045] performing binarization processing on the third region image to obtain a fourth region image;

[0046] Performing a binarization result detection on the fourth region image to obtain a binarization detection result;

[0047] determining whether the radar image meets the quality threshold based on the binarization detection result, wherein if the binarization result is a first result, determining that the radar image meets the quality threshold, and if the binarization result is a second result, determining that the radar image does not meet the quality threshold;

[0048] Determine a second number of data groups included in the second initial data set whose quality satisfies the above-mentioned predetermined quality threshold;

[0049] Whether the second initial data set satisfies the second condition is determined based on the second quantity, wherein, when the second quantity is greater than or equal to a quantity threshold, it is determined that the second initial data set satisfies the second condition, and when the second quantity is less than the quantity threshold, it is determined that the second initial data set does not satisfy the second condition.

[0050] In an exemplary embodiment, determining whether the second initial data set satisfies the second condition includes:

[0051] Acquire the number of data groups included in the second initial data set, and acquire the quality of each data group included in the second initial data set;

[0052] Whether the second initial data set satisfies the second condition is determined based on the quantity and the quality, wherein, when it is determined that the quantity satisfies the quantity condition and the quality satisfies the quality condition, it is determined whether the second initial data set satisfies the second condition, and when it is determined that the quantity does not satisfy the quantity condition and / or the quality does not satisfy the quality condition, it is determined that the first initial data set does not satisfy the second condition.

[0053] In an exemplary embodiment, after determining whether the second initial data set satisfies the second condition based on the quantity and the quality, the method further includes:

[0054] In the case where it is determined that the quantity does not satisfy the quantity condition, adding a data group included in the second initial data set;

[0055] In the case where it is determined that the quality does not satisfy the quality condition, all data groups included in the second data set are deleted, and the data groups included in the second data set are re-determined.

[0056] According to another embodiment of the present invention, there is provided an image calibration device, comprising:

[0057] An image acquisition module, used to acquire an initial pixel image and an initial radar image of a target scene, wherein the initial pixel image includes pixel information of a target object contained in the target scene, the initial radar image includes radar information of the target object, and the initial pixel image and the initial radar image are obtained by synchronously shooting the target scene;

[0058] a first calibration module, configured to perform a first calibration process on the initial pixel image to obtain a target pixel image, wherein the target pixel image is a calibration image of the initial pixel image, and the target pixel image includes the initial pixel image and radar information of the target object;

[0059] a second calibration module, configured to perform a second calibration process on the initial radar image to obtain a target radar image, wherein the target radar image is a calibration image of the initial radar image, and the target radar image includes pixel information of the initial radar image and the target object;

[0060] A fusion module is used to perform a fusion operation on the target radar image and the target pixel image to obtain a target data image, wherein the target data image includes the target radar image and the target pixel image.

[0061] According to yet another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any one of the above method embodiments when run.

[0062] According to yet another embodiment of the present invention, there is provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0063] Through the present invention, since the pixel image calibrated with radar information is fused with the radar image calibrated with pixel information, the coordinate information of the target object in the target scene can be intuitively understood, so that the movement trend of the target object in the target scene can be judged. Therefore, the problems of poor recognition accuracy and low recognition efficiency in the related technology can be solved, and the effect of improving recognition accuracy and efficiency can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 is a hardware structure block diagram of a mobile terminal of an image calibration method according to an embodiment of the present invention;

[0065] Figure 2 is a flow chart of an image calibration method according to an embodiment of the present invention;

[0066] Figure 3 is a schematic diagram of an ideal scene in an image calibration method according to an embodiment of the present invention;

[0067] Figure 4 is a schematic diagram of a non-ideal scene in an image calibration method according to an embodiment of the present invention;

[0068] Figure 5 is a structural block diagram of an image calibration device according to an embodiment of the present invention;

[0069] Figure 6 is a schematic diagram of data to be quantitatively and qualitatively evaluated according to a specific embodiment of the present invention;

[0070] Figure 7 is a schematic diagram of data for performing target area extraction according to a specific embodiment of the present invention;

[0071] Figure 8 is a schematic diagram of data for performing downsampling according to a specific embodiment of the present invention;

[0072] Fig. 9 is a schematic diagram of data for performing image quality detection according to a specific embodiment of the present invention. Figure 1 ;

[0073] Fig.10 is a schematic diagram of data for performing image quality detection according to a specific embodiment of the present invention. Figure 2 ;

[0074] Fig.11 is a complete flow chart according to a specific embodiment of the present invention;

[0075] Fig.12 is a pixel map for performing a calibration operation according to a specific embodiment of the present invention;

[0076] Fig.13 is a radar chart showing a calibration operation according to a specific embodiment of the present invention. DETAILED DESCRIPTION

[0077] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings and in combination with the embodiments.

[0078] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0079] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1FIG. 1 is a hardware structure block diagram of a mobile terminal of an image calibration method according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0080] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as a computer program corresponding to an image calibration method in an embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0081] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0082] In this embodiment, an image calibration method is provided. Figure 2 is a flow chart of an image calibration method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0083] Step S202, acquiring an initial pixel image and an initial radar image of the target scene, wherein the initial pixel image includes pixel information of a target object contained in the target scene, the initial radar image includes radar information of the target object, and the initial pixel image and the initial radar image are obtained by synchronously photographing the target scene;

[0084] In this embodiment, the initial pixel image is obtained in order to determine information such as the type of the target object in the target scene, and the initial radar image is obtained in order to determine information such as the position and movement direction of the target object in the target scene; and the synchronous shooting is to reduce errors in the calibration process, thereby improving the calibration accuracy.

[0085] Among them, the initial pixel image can be (but not limited to) a picture or a video. The number of pictures can be one frame, thereby reducing the amount of calculation, or multiple frames, thereby improving the accuracy of image calibration; the initial pixel image can be (but not limited to) an infrared image, a visible light image, or a fusion of the two; the acquisition of the initial pixel image can be (but not limited to) obtained by collecting the target area through a visible light camera or a visible light sensor, or by collecting the target area through an infrared camera or an infrared sensor; the initial radar image can be (but not limited to) collected by a radar sensor from the target scene; the target object can be (but not limited to) pedestrians, buildings (such as pools, railings, roads), vehicles, animals, For plants and other things, the pixel information of the target object may be (but not limited to) the number, type, position, type, color, size and other information of the pixel of the target object; the radar information includes the spatial position, movement speed and other information of the target object; the synchronous shooting may be (but not limited to) realized by synchronizing the image acquisition device, or by synchronizing the acquired initial pixel image and the initial radar image according to the timestamp, or by other processing methods; the target scene may be an ideal scene that meets the preset conditions, wherein the ideal environment refers to an environment in which there are very sparse target objects, the measurement quality of the target objects is good, and the radar video target matching is unambiguous, for example, a target scene that meets the following conditions:

[0086] a) Video targets and radar targets are very sparse to ensure no ambiguity;

[0087] b) The number of video targets and radar targets is the same;

[0088] c) There is no occluded part of the video target;

[0089] d) There is no sudden change in radar target speed;

[0090] e) pedestrian targets;

[0091] f) The video target does not appear to be in the air;

[0092] g) The image is well-lit (not at night).

[0093] It should be noted that the purpose of condition a is to ensure that the association between video targets and radar targets is unambiguous. When there are many video targets or radar targets, there are many possible association situations. The purpose of condition b is also a means to ensure unambiguity, in which the number of radar targets and the number of video targets must be the same; condition c is to ensure the quality of image data. Figure 3 and Figure 4 As shown, Figure 3 is a person's full body frame, and Figure 4 Since the occlusion is a half-body frame, when extracting a certain point of the frame, such as the center point of the lower edge, the video target point sequence will undergo a large change at the occlusion, that is, an uncontrollable error without any mathematical model will be introduced. At this time, the measurement can be regarded as an incorrect target measurement. If such a measurement is introduced into the network for training, it will cause the network to not converge or become unstable.

[0094] Condition d is to ensure the quality of radar data. Since radar is particularly sensitive to moving targets, it is possible to determine whether a target is a normal target or an abnormal target based on the target speed. Generally, under normal circumstances, the speed of a pedestrian will not change suddenly; if the speed changes suddenly, the target displayed on the radar may not be a pedestrian, but something else, such as a cat or dog.

[0095] Condition e is to filter invalid targets. The video intelligence algorithm can identify some specified target types, such as pedestrians, non-motor vehicles, motor vehicles, cats, dogs, etc. If there are non-motor vehicles or motor vehicles, cats, and dogs in the environment, they need to be filtered. In this embodiment, pedestrians are used as the data source for training and testing.

[0096] Condition f is to ensure data quality. Since the video image acquisition device cannot obtain the spatial depth information of the target object, if the target object is vacant, the one-to-one mapping relationship between the radar and the image will be destroyed, which will directly lead to the failure of detection. Therefore, it is necessary to determine through condition f that the target object is not vacant.

[0097] Condition g is to ensure the stability of the acquisition frame used to identify the target object in the pixel image. When the target scene is in an environment with poor lighting conditions such as at night, the edge of the target object will be blurred due to dim light and other reasons. At this time, the acquisition frame fluctuates greatly, which can easily lead to unstable video acquisition. Therefore, it is necessary to ensure the lighting conditions in the pixel image through condition g.

[0098] Step S204, performing a first calibration process on the initial pixel image to obtain a target pixel image, wherein the target pixel image is a calibrated image of the initial pixel image, and the target pixel image includes the initial pixel image and radar information of the target object;

[0099] In this embodiment, the initial pixel image is calibrated to obtain radar information of the target object in the initial pixel image, so that information such as the moving speed of the target object can be determined.

[0100] Among them, the calibration of the initial pixel image can be achieved (but not limited to) through a neural network, or through a preset algorithm, or through a combination of a preset algorithm and a neural network; the first calibration process can be to mark and display the position of the target object and radar information through an identification frame in the initial pixel image, or to recolor the target object with a preset color, and then mark the corresponding radar information in the area of ​​the target object to achieve calibration of the target object.

[0101] For example, the initial pixel image is input into a trained pixel mapping model, and then the pixel mapping model maps the target object contained in the initial pixel image to the initial radar image, and matches the radar information of the target object in the initial radar image according to the mapping relationship between the initial pixel image and the initial radar image, so as to calibrate the radar information in the area where the target object is located in the initial pixel image, and then use the initial pixel image calibrated with the radar information as the target pixel image.

[0102] It should be noted that when calibrating the initial pixel image, a time series can be established by selecting a suitable point in the target frame:

[0103] t i , BoxPt i

[0104] t i+1 , BoxPt i+1

[0105] t i+2 , BoxPt i+2

[0106] …

[0107] where t i Indicates time, BoxPt i =(u i ,v i )=U i Indicates the position of a certain point in the video target frame.

[0108] Step S206, performing a second calibration process on the initial radar image to obtain a target radar image, wherein the target radar image is a calibrated image of the initial radar image, and the target radar image includes pixel information of the initial radar image and the target object;

[0109] In this embodiment, the initial radar image is calibrated to obtain pixel information of the target object in the initial radar image, so that the type, spatial size and other information of the target object can be determined.

[0110] Among them, the calibration of the initial radar image can be (but not limited to) implemented through a neural network, or it can be implemented through a preset algorithm, or it can be implemented through a combination of a preset algorithm and a neural network; the second calibration processing can be to mark and display the position and pixel information of the target object in the initial radar image through an identification frame, or it can be to recolor the representative points represented by the target object with a preset color, and then mark the corresponding pixel information in the area of ​​the target object to realize the calibration of the target object.

[0111] For example, the initial radar image is input into a trained radar mapping model, and then the radar mapping model maps the target object contained in the initial radar image to the initial pixel image, and matches the pixel information of the target object in the initial pixel image according to the mapping relationship between the initial pixel image and the initial radar image, so as to calibrate the pixel information in the initial radar image at the area where the target object is located, and then use the initial radar image with the calibrated pixel information as the target radar image.

[0112] It should be noted that when calibrating the initial radar image, the following time series can be established:

[0113] By detecting and tracking the number and position of pedestrians, the following time series can be established

[0114] t i , CartPt i

[0115] t i+1 , CartPt i+1

[0116] t i+2 , CartPt i+2

[0117] …

[0118] Among them, CartPt i =(x i ,y i )=X i Indicates a radar target point.

[0119] Then, according to the time series of the initial pixel image and the time series of the initial radar image, a video radar measurement data set processed by the matching algorithm can be obtained, namely:

[0120] t i , BoxPt i , CartPt i

[0121] t i+1 , BoxPt i+1 , CartPt i+1

[0122] t i+2 , BoxPt i+2 , CartPt i+2

[0123] Based on the above data set, the following mapping relationship can be obtained:

[0124] U =F(X)(1)

[0125] X=G(U) (2)

[0126] F=G -1 (3)

[0127] G =F -1 (4)

[0128] Among them, F(*) represents mapping radar coordinates to pixel information one by one, and G(*) represents mapping pixel information in the pixel image to radar coordinates one by one.

[0129] Step S208, performing a fusion operation on the target radar image and the target pixel image to obtain a target data image, wherein the target data image includes the target radar image and the target pixel image.

[0130] In this embodiment, the target radar image and the target pixel image are fused in order to be able to observe the radar image and the pixel image simultaneously in the same visualization interface, so as to more accurately identify and judge the state of the target object in the target scene.

[0131] Among them, the fusion operation can be (but not limited to) displaying the target radar image and the target pixel image side by side in the same visualization interface in the same size, or displaying them side by side in the same visualization interface in a large and small manner, or displaying the larger image in the center of the visualization interface and the smaller image at the edge of the visualization interface (for example, the lower right corner) in a large and small manner; the fusion operation can also be to capture the target object in one of the images and paste it to the corresponding position in the other image, or other fusion methods.

[0132] Through the above steps, since the pixel image calibrated with radar information is fused with the radar image calibrated with pixel information, the coordinate information of the target object in the target scene can be intuitively understood, so that the movement trend of the target object in the target scene can be judged. Therefore, the problems of poor recognition accuracy and low recognition efficiency in related technologies can be solved, and the image accuracy and efficiency are improved.

[0133] The execution subject of the above steps may be a base station, a terminal, etc., but is not limited thereto.

[0134] The order of the above-mentioned step S204 and step S206 can be exchanged, that is, step S206 is executed first, and then step S204 is executed.

[0135] In an optional embodiment, before acquiring the initial pixel image and the initial radar image of the target scene, the method includes:

[0136] Step S2002, performing scene recognition on the target scene to determine whether the target scene meets the first condition;

[0137] Obtain the initial pixel image and initial radar image of the target scene, including:

[0138] Step S2022: When the target scene satisfies the first condition, an initial pixel image and an initial radar image of the target scene are acquired.

[0139] In this embodiment, scene recognition is performed on the target scene in order to determine that the target scene is an ideal scene, so that the quality of the collected initial radar image and initial pixel image meets the requirements, reduce interference in the image calibration process, and thus improve the calibration efficiency and calibration accuracy.

[0140] Among them, the execution of scene recognition can be achieved through built-in CPU, FPGA and other logical operation modules, or through external computers and cloud processing terminals. When the scene device is implemented through an external device, its data transmission can be achieved through Ethernet, or through wireless transmission networks such as 3G / 4G / 5G, or through other methods; the first condition can be any one of the aforementioned conditions ag, or a combination of any multiple of them, or a condition other than the aforementioned conditions ag.

[0141] In an optional embodiment, performing a first calibration process on the initial pixel image to obtain a target pixel image includes:

[0142] Step S2042, using the first model to perform a first calibration process on the initial pixel image to obtain a target pixel image, wherein the first model is trained through machine learning using a first data set, and the first data set includes multiple groups of first data, and each group of first data includes: a pixel image and a calibrated image of the pixel image.

[0143] In this embodiment, using the first model to calibrate the initial pixel image can reduce the amount of manual processing, thereby reducing errors, and also reduce the workload of calibration work, thereby improving image recognition accuracy and efficiency.

[0144] Among them, the first model can be (but not limited to) a neural network model trained by machine learning using the first data set, or a model combination of multiple neural network models trained by machine learning using the first data set; in addition to the first data, the first data set can also (but not limited to) include other information, such as the identification and storage path of the first data; in addition to the pixel image and the calibrated image of the pixel image, the first data can also (but not limited to) include the identification information, image size and other information of the pixel image; the pixel image and the calibrated image of the pixel image in each group of the first data can be only one pair or multiple pairs. The calibrated image of the pixel image can be a pixel image with the radar information of the target object calibrated, or a pixel image with the radar information and other information calibrated; the machine learning training can be a neural network learning training, or it can be a process of manually revising the model.

[0145] In an optional embodiment, before performing the first calibration process on the initial pixel image, the method further includes:

[0146] Step S20402, obtaining a first initial data set, wherein the first initial data set includes multiple groups of first data, and each group of first data includes a pixel image and a calibration image of the pixel image;

[0147] Step S20404, determining whether the first initial data set meets the second condition;

[0148] Step S20406: When the first initial data set meets the second condition, determine the first initial data set as the first data set.

[0149] In this embodiment, judging whether the first initial data set satisfies the second condition is to determine whether the data used to train the first model meets the requirements, thereby ensuring that the trained first model can accurately calibrate the initial pixel image, reduce calibration errors, and thereby improve image recognition efficiency and recognition accuracy.

[0150] The first initial data set may be (but not limited to) image data of a target scene acquired by an image acquisition device, preset image data, image data of other scenes acquired by an image acquisition device, or automatically generated by a preset algorithm; the second condition may be (but not limited to) whether the amount of image data reaches a threshold, and whether the quality of the image data reaches a threshold, or other content; the judgment of whether the second condition is satisfied may be by judging the quality of the image data in the first initial data set (for example, the distribution of pixels), or by judging the amount of image data in the first initial data set, or by judging the comprehensive result of the quality and quantity of the image data in the first initial data set, or other forms of judgment; the judgment of whether the second condition is satisfied may be realized by built-in devices or modules such as CPU and FPGA, or by a connected external computer or cloud computing server, wherein the data transmission method may be realized by Ethernet, or by a wireless communication network connected to an external network; the first initial data set may be (but not limited to) only one, or multiple.

[0151] In an optional embodiment, determining whether the first initial data set satisfies the second condition includes:

[0152] Step S204042, performing the following operations on each set of data included in the first initial data set to determine a set of data included in the first initial data set whose quality meets a predetermined quality threshold:

[0153] Step S2040422, extracting the first target area image contained in the pixel image;

[0154] Step S2040424, downsampling the first target region image to obtain a first region image;

[0155] Step S2040426, performing binarization processing on the first region image to obtain a second region image;

[0156] Step S2040428, performing a binarization result detection on the second region image to obtain a binarization detection result;

[0157] Step S20404210, judging whether the pixel image meets the quality threshold based on the binarization detection result, wherein when the binarization result is a first result, it is determined that the pixel image meets the quality threshold, and when the binarization result is a second result, it is determined that the pixel image does not meet the quality threshold;

[0158] Step S204044, determining a first number of data groups included in the first initial data set whose quality meets the above-mentioned predetermined quality threshold;

[0159] Step S204046, based on the first quantity, determine whether the first initial data set meets the second condition, wherein, when the first quantity is greater than or equal to the quantity threshold, it is determined that the first initial data set meets the second condition, and when the first quantity is less than the quantity threshold, it is determined that the first initial data set does not meet the second condition.

[0160] In this embodiment, firstly performing quality judgment on pixel images and then performing quantity judgment on pixel images that meet the quality requirements can reduce the amount of calculation in the judgment process and save the time consumed by the judgment process.

[0161] Among them, extracting the first target area image can be (but not limited to) achieved by intercepting the target area along the edge of the shape according to a predetermined shape, or by intercepting different positions of the target area according to the predetermined shape and then splicing them, or it can be extracted by other methods; the first target area image can be one or more images; downsampling processing is to reduce the pixels of the first target area image to facilitate binarization processing, wherein before the binarization processing, the first area image after pixel reduction can also be grid processed, that is, adding a grid on the first area image to segment the first area image, so as to facilitate binarization processing; binarization processing can (but not limited to) be marking the area that meets the pixel requirements as 1 or 0, or marking the area that meets the pixel requirements with two different colors, such as white and black, or white and green, etc.; binarization result detection can (but not limited to) be detecting the uniformity of binary distribution and the proportion of a certain value after binarization.

[0162] For example, when a pedestrian passes through an area in the binary image, the corresponding value of the area is set to 1, and then the area with a value of 1 and the proportion of 1 are detected; when the distribution of the areas with a value of 1 is detected to be relatively uniform and the proportion of 1 is greater than or equal to 40%, it is judged that the quality of the image meets the threshold requirement.

[0163] In an optional embodiment, determining whether the first initial data set satisfies the second condition includes:

[0164] Step S204048, obtaining the number of data groups included in the first initial data set, and obtaining the quality of each data group included in the first initial data set;

[0165] Step S2040410, judging whether the first initial data set satisfies the second condition based on the quantity and the quality, wherein, when it is determined that the quantity satisfies the quantity condition and the quality satisfies the quality condition, determining whether the first initial data set satisfies the second condition, and when it is determined that the quantity does not satisfy the quantity condition and / or the quality does not satisfy the quality condition, determining that the first initial data set does not satisfy the second condition.

[0166] In this embodiment, after obtaining the number of data groups included in the first initial data set and the quality of each data group included in the first initial data set, the quantity and quality are directly judged, which reduces the computational complexity of the judgment process and improves image recognition efficiency.

[0167] Among them, the number of data groups included in the first initial data set and the quality of each data group included in the first initial data set can be achieved through a built-in quantity statistics module and a quality detection module, or can be achieved through an external computing server; the acquisition method can be obtained by data transmission through Ethernet, or can be obtained by data transmission through a wireless network or other methods; the number of data groups included in the first initial data set and the quality of each data group included in the first initial data set can be obtained sequentially or simultaneously.

[0168] In an optional embodiment, after determining whether the first initial data set meets the second condition based on the quantity and quality, the method further includes:

[0169] Step S20408, when it is determined that the quantity does not satisfy the quantity condition, adding a data group included in the first initial data set;

[0170] Step S204010: When it is determined that the quality does not meet the quality condition, all data groups included in the first data set are deleted, and the data groups included in the first data set are re-determined.

[0171] In this embodiment, when the quantity meets the requirements, the data groups included in the first initial data set are continuously collected and supplemented to avoid model training failure due to insufficient quantity; when the quality does not meet the requirements, all data groups are deleted and the data groups are re-collected and filled to avoid the impact of images that do not meet the quality requirements on model training.

[0172] The increase of the data group included in the first initial data set may be achieved by continuously collecting data from the target scene, or by continuing to input data, or by automatically generating data according to a preset algorithm.

[0173] In an optional embodiment, performing a second calibration process on the initial radar image to obtain a target radar image includes:

[0174] Step S2062: Perform a second calibration process on the initial radar image using the second model to obtain a target radar image, wherein the second model is trained by machine learning using a second data set, and the second data set includes multiple groups of second data, each group of second data includes: a radar image and a calibrated image of the radar image.

[0175] In this embodiment, using the second model to calibrate the initial radar image can reduce the amount of manual processing, thereby reducing errors, and also reduce the workload of calibration work, thereby improving image recognition accuracy and efficiency.

[0176] The second model may be (but not limited to) a neural network model trained by machine learning using the second data set, or a model combination of multiple neural network models trained by machine learning using the second data set; the second data set may include (but not limited to) other information in addition to the second data, such as the identification and storage path of the second data; the second data may include (but not limited to) identification information of the radar image, radar point size and other information in addition to the radar image and the calibrated image of the radar image; the radar image and the calibrated image of the pixel image in each set of the second data may be only one pair or multiple pairs. The calibrated image of the radar image may be a radar image with pixel information of the target object calibrated, or a radar image with pixel information and other information calibrated; the machine learning training may be a neural network learning training, or a process of manually revising the model.

[0177] In an optional embodiment, before performing the second calibration process on the initial radar image, the method further includes:

[0178] Step S20602, obtaining a second initial data set, wherein the second initial data set includes multiple sets of second data, and each set of second data includes a radar image and a calibration image of the radar image;

[0179] Step S20604, determining whether the second initial data set meets the second condition;

[0180] Step S20606: If the second initial data set meets the second condition, determine the second initial data set as the second data set.

[0181] In this embodiment, judging whether the second initial data set satisfies the second condition is to determine whether the data used to train the second model meets the requirements, thereby ensuring that the trained second model can accurately calibrate the initial radar image, reduce calibration errors, and thereby improve image recognition efficiency and recognition accuracy.

[0182] The second initial data set may be (but not limited to) image data of a target scene acquired by an image acquisition device, preset image data, image data of other scenes acquired by an image acquisition device, or automatically generated by a preset algorithm; the second condition may be (but not limited to) whether the amount of image data reaches a threshold, and whether the quality of the image data reaches a threshold, or other content; the judgment of whether the second condition is satisfied may be by judging the quality of the image data in the second initial data set (for example, the distribution of radar points), or by judging the amount of image data in the second initial data set, or by judging the comprehensive result of the quality and quantity of the image data in the second initial data set, or other forms of judgment; the judgment of whether the second condition is satisfied may be implemented by built-in devices or modules such as CPU and FPGA, or by a connected external computer or cloud computing server, wherein the data transmission method may be implemented by Ethernet, or by a wireless communication network connected to an external network; the first initial data set may be (but not limited to) only one, or multiple.

[0183] In an optional embodiment, determining whether the second initial data set satisfies the second condition includes:

[0184] Step S206042, performing the following operations on each set of data included in the second initial data set to determine a set of data included in the second initial data set whose quality meets a predetermined quality threshold:

[0185] Step S2060422, extracting a second target area image contained in the radar image;

[0186] Step S2060424, downsampling the second target region image to obtain a third region image;

[0187] Step S2060426, performing binarization processing on the third region image to obtain a fourth region image;

[0188] Step S2060428, performing a binarization result detection on the fourth region image to obtain a binarization detection result;

[0189] Step S20604210, judging whether the radar image meets the quality threshold based on the binarization detection result, wherein when the binarization result is a first result, it is determined that the radar image meets the quality threshold, and when the binarization result is a second result, it is determined that the radar image does not meet the quality threshold;

[0190] Step S206044, determining a second number of data groups included in the second initial data set whose quality meets the above-mentioned predetermined quality threshold;

[0191] Step S206046, judging whether the second initial data set satisfies a second condition based on a second quantity, wherein, when the second quantity is greater than or equal to a quantity threshold, determining that the second initial data set satisfies the second condition, and when the second quantity is less than the quantity threshold, determining that the second initial data set does not satisfy the second condition.

[0192] In this embodiment, firstly performing a quality judgment on the radar image and then performing a quantity judgment on the pixel images that meet the quality requirements can reduce the amount of calculation in the judgment process and save the time consumed by the judgment process.

[0193] The second target area image can be extracted (but not limited to) by intercepting the target area along the edge of the shape according to a predetermined shape, or by intercepting different positions of the target area according to the predetermined shape and then splicing them, or by other methods; the second target area image can be one or more images; the downsampling process is to reduce the pixels of the second target area image to facilitate binarization, wherein before the binarization process, the third area image after the pixel reduction can also be grid processed, that is, a grid is added to the third area image to segment the third area image, so as to facilitate the binarization process; the binarization process can be (but not limited to) marking the area that meets the radar requirements as 1 or 0, or marking the area that meets the radar requirements with two different colors, such as white and black, or white and green, etc.; the binarization result detection can be (but not limited to) detecting the uniformity of the binary distribution and the proportion of a certain value after binarization.

[0194] For example, when a pedestrian passes through an area in the binary image, the corresponding value of the area is set to 1, and then the area with a value of 1 and the proportion of 1 are detected; when the distribution of the areas with a value of 1 is detected to be relatively uniform and the proportion of 1 is greater than or equal to 40%, it is judged that the quality of the image meets the threshold requirement.

[0195] In an optional embodiment, determining whether the second initial data set satisfies the second condition includes:

[0196] Step S206048, obtaining the number of data groups included in the second initial data set, and obtaining the quality of each data group included in the first initial data set;

[0197] Step S2060410, judging whether the second initial data set satisfies the second condition based on the quantity and the quality, wherein, when it is determined that the quantity satisfies the quantity condition and the quality satisfies the quality condition, determining whether the second initial data set satisfies the second condition; when it is determined that the quantity does not satisfy the quantity condition and / or the quality does not satisfy the quality condition, determining that the second initial data set does not satisfy the second condition.

[0198] In this embodiment, after obtaining the number of data groups included in the second initial data set and the quality of each data group included in the second initial data set, the quantity and quality are directly judged, which reduces the computational complexity of the judgment process and improves image recognition efficiency.

[0199] Among them, the number of data groups included in the second initial data set and the quality of each data group included in the second initial data set can be achieved through a built-in quantity statistics module and a quality detection module, or can be achieved through an external computing server; the acquisition method can be obtained by data transmission through Ethernet, or can be obtained by data transmission through a wireless network or other methods; the number of data groups included in the second initial data set and the quality of each data group included in the second initial data set can be obtained sequentially or simultaneously.

[0200] In an optional embodiment, after determining whether the second initial data set meets the second condition based on the quantity and quality, the method further includes:

[0201] Step S2064, when it is determined that the quantity does not satisfy the quantity condition, adding a data group included in the second initial data set;

[0202] Step S2068: When it is determined that the quality does not meet the quality condition, all data groups included in the second data set are deleted, and the data groups included in the second data set are re-determined.

[0203] In this embodiment, when the quantity meets the requirements, the data groups included in the second initial data set are continuously collected and supplemented to avoid model training failure due to insufficient quantity; when the quality does not meet the requirements, all data groups are deleted and the data groups are re-collected and filled to avoid the impact of images that do not meet the quality requirements on model training.

[0204] The increase of the data group included in the second initial data set may be achieved by continuously collecting data from the target scene, or by continuing to input data, or by automatically generating data according to a preset algorithm.

[0205] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0206] In this embodiment, an image calibration device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0207] Figure 5 is a structural block diagram of a space calibration device according to an embodiment of the present invention. Figure 5 As shown, the device comprises:

[0208] An image acquisition module 32 is used to acquire an initial pixel image and an initial radar image of a target scene, wherein the initial pixel image includes pixel information of a target object contained in the target scene, the initial radar image includes radar information of the target object, and the initial pixel image and the initial radar image are obtained by synchronously photographing the target scene;

[0209] A first calibration module 34 is used to perform a first calibration process on the initial pixel image to obtain a target pixel image, wherein the target pixel image is a calibration image of the initial pixel image, and the target pixel image includes the initial pixel image and radar information of the target object;

[0210] A second calibration module 36 is used to perform a second calibration process on the initial radar image to obtain a target radar image, wherein the target radar image is a calibration image of the initial radar image, and the target radar image includes pixel information of the initial radar image and the target object;

[0211] The fusion module 38 is used to perform a fusion operation on the target radar image and the target pixel image to obtain a target data image, wherein the target data image includes the target radar image and the target pixel image.

[0212] In an optional embodiment, the device further comprises:

[0213] A scene recognition module 30, used to perform scene recognition on the target scene before acquiring the initial pixel image and the initial radar image of the target scene to determine whether the target scene meets the first condition;

[0214] The image acquisition module 32 includes:

[0215] The image acquisition unit 322 is used to acquire an initial pixel image and an initial radar image of the target scene when the target scene meets the first condition.

[0216] In an optional embodiment, the first calibration module 34 includes:

[0217] The first calibration unit 342 is used to perform a first calibration process on the initial pixel image using a first model to obtain a target pixel image, wherein the first model is trained by machine learning using a first data set, and the first data set includes multiple groups of first data, and each group of first data includes: a pixel image and a calibrated image of the pixel image.

[0218] In an optional embodiment, the first calibration module 34 further includes:

[0219] A first initial data acquisition unit 344 is used to obtain a first initial data set before performing a first calibration process on the initial pixel image, wherein the first initial data set includes multiple groups of first data, each group of first data includes a pixel image and a calibration image of the pixel image;

[0220] A first initial determination unit 346, configured to determine whether the first initial data set satisfies a second condition;

[0221] The first data set determining unit 348 is configured to determine the first initial data set as the first data set if the first initial data set meets the second condition.

[0222] In an optional embodiment, the first initial determination unit 346 includes:

[0223] The first quality judgment subunit 3462 is configured to perform the following operations on each set of data included in the first initial data set to determine the data set included in the first initial data set whose quality meets a predetermined quality threshold:

[0224] Extracting a first target area image contained in the pixel image;

[0225] Downsampling the first target region image to obtain a first region image;

[0226] Binarizing the first region image to obtain a second region image;

[0227] Performing a binarization result detection on the second region image to obtain a binarization detection result;

[0228] Determining whether the pixel image meets the quality threshold based on the binarization detection result, wherein when the binarization result is a first result, it is determined that the pixel image meets the quality threshold, and when the binarization result is a second result, it is determined that the pixel image does not meet the quality threshold;

[0229] The first quantity determination subunit 3464 is used to determine the first quantity of data groups included in the first initial data set whose quality meets the above-mentioned predetermined quality threshold;

[0230] The first condition judgment subunit 3466 is used to judge whether the first initial data set meets the second condition based on the first quantity, wherein when the first quantity is greater than or equal to the quantity threshold, it is determined that the first initial data set meets the second condition; when the first quantity is less than the quantity threshold, it is determined that the first initial data set does not meet the second condition.

[0231] In an optional embodiment, the first initial determination unit 346 further includes:

[0232] The first data quality extraction subunit 3468 is used to obtain the number of data groups included in the first initial data set, and to obtain the quality of each data group included in the first initial data set;

[0233] The second condition judgment subunit 34610 is used to judge whether the first initial data set meets the second condition based on quantity and quality, wherein, when it is determined that the quantity meets the quantity condition and the quality meets the quality condition, it is determined whether the first initial data set meets the second condition; when it is determined that the quantity does not meet the quantity condition and / or the quality does not meet the quality condition, it is determined that the first initial data set does not meet the second condition.

[0234] In an optional embodiment, the first calibration module 34 further includes:

[0235] A first data expansion unit 3410 is configured to, after determining whether the first initial data set meets a second condition based on the quantity and quality, and if it is determined that the quantity does not meet the quantity condition, add a data group included in the first initial data set;

[0236] The first data recollection unit 3412 is used to delete all data groups included in the first data set and redetermine the data groups included in the first data set when it is determined that the quality does not meet the quality condition.

[0237] In an optional embodiment, the second calibration module 36 includes:

[0238] The second calibration unit 362 is used to perform a second calibration process on the initial radar image using a second model to obtain a target radar image, wherein the second model is trained by machine learning using a second data set, and the second data set includes multiple groups of second data, each group of second data includes: a radar image and a calibrated image of the radar image.

[0239] In an optional embodiment, the second calibration module 36 further includes:

[0240] A second initial data acquisition unit 364 is used to obtain a second initial data set before performing a second calibration process on the initial radar image, wherein the second initial data set includes a plurality of groups of second data, each group of second data includes a radar image and a calibration image of the radar image;

[0241] A second initial determination unit 366, configured to determine whether the second initial data set satisfies a second condition;

[0242] The second data set determining unit 368 is configured to determine the second initial data set as the second data set if the second initial data set meets the second condition.

[0243] In an optional embodiment, the second initial determination unit 366 includes:

[0244] The second quality judgment subunit 3662 is configured to perform the following operations on each set of data included in the second initial data set to determine a set of data included in the second initial data set whose quality meets a predetermined quality threshold:

[0245] Extracting a second target area image contained in the radar image;

[0246] Downsampling the second target region image to obtain a third region image;

[0247] Binarizing the third region image to obtain a fourth region image;

[0248] Performing a binarization result detection on the fourth region image to obtain a binarization detection result;

[0249] Determining whether the radar image meets a quality threshold based on the binarization detection result, wherein if the binarization result is a first result, determining that the radar image meets the quality threshold, and if the binarization result is a second result, determining that the radar image does not meet the quality threshold;

[0250] The second quantity determination subunit 3664 is used to determine the second quantity of data groups included in the second initial data set whose quality meets the above-mentioned predetermined quality threshold;

[0251] The third condition judgment subunit 3666 judges whether the second initial data set satisfies the second condition based on the second quantity, wherein, when the second quantity is greater than or equal to the quantity threshold, it is determined that the second initial data set satisfies the second condition; when the second quantity is less than the quantity threshold, it is determined that the second initial data set does not satisfy the second condition.

[0252] In an optional embodiment, the second initial determination unit 366 further includes:

[0253] The second data quality extraction subunit 3668 is used to obtain the number of data groups included in the second initial data set, and to obtain the quality of each data group included in the first initial data set;

[0254] The fourth condition judgment subunit 36610 judges whether the second initial data set meets the second condition based on the quantity and the quality, wherein, if it is determined that the quantity meets the quantity condition and the quality meets the quality condition, it is determined whether the second initial data set meets the second condition, and if it is determined that the quantity does not meet the quantity condition and / or the quality does not meet the quality condition, it is determined that the first initial data set does not meet the second condition

[0255] In an optional embodiment, the second calibration module 36 further includes:

[0256] A second data expansion unit 3610 is used to, after determining whether the second initial data set meets the second condition based on the quantity and quality, and if it is determined that the quantity does not meet the quantity condition, add a data group included in the second initial data set;

[0257] The second data recollection unit 3612 is used to delete all data groups included in the first data set and redetermine the data groups included in the first data set when it is determined that the quality does not meet the quality condition.

[0258] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0259] The present invention is described below in conjunction with specific embodiments.

[0260] Before calibration, the scene must be identified to determine whether the target scene is an ideal environment, where an ideal environment is a prerequisite for selecting training data and test data. In this embodiment, an ideal environment refers to an environment where there are very sparse targets, the target measurement quality is good, and the radar video target matching is unambiguous, and the ideal environment needs to meet the following conditions:

[0261] a) Video targets and radar targets are very sparse to ensure no ambiguity;

[0262] b) The number of video targets and radar targets is the same;

[0263] c) There is no occluded part of the video target;

[0264] d) There is no sudden change in radar target speed;

[0265] e) pedestrian targets;

[0266] f) The video target does not appear to be in the air;

[0267] g) The image is well-lit (not at night).

[0268] It should be noted that the purpose of condition a is to ensure that the association between video targets and radar targets is unambiguous. When there are many video targets or radar targets, there are many possible association scenarios. The purpose of condition b is also a means to ensure unambiguity, in which the number of radar targets and the number of video targets must be the same; condition c is to ensure the quality of image data. As mentioned above Figure 3 and Figure 4 As shown, Figure 3 is a person's full body frame, and Figure 4 It is the half-body frame that is blocked. When extracting a certain point of the frame, such as the center point of the lower edge, the sequence of video target points will change drastically at the blocked part, which will introduce uncontrollable errors without any mathematical model. At this time, the measurement can be regarded as incorrect target measurement. If such measurement is introduced into the network for training, it will cause the network to not converge or become unstable.

[0269] Condition d is to ensure the quality of radar data. Since radar is particularly sensitive to moving targets, it is possible to determine whether a target is a normal target or an abnormal target based on the target speed. Generally, under normal circumstances, the speed of a pedestrian will not change suddenly; if the speed changes suddenly, the target displayed on the radar may not be a pedestrian, but something else, such as a cat or dog.

[0270] Condition e is to filter invalid targets. The video intelligence algorithm can identify some specified target types, such as pedestrians, non-motor vehicles, motor vehicles, cats, dogs, etc. If there are non-motor vehicles or motor vehicles, cats, and dogs in the environment, they need to be filtered. In this embodiment, pedestrians are used as the data source for training and testing.

[0271] Condition f is to ensure data quality. Since the video image acquisition device cannot obtain the spatial depth information of the target object, if the target object is vacant, the one-to-one mapping relationship between the radar and the image will be destroyed, which will directly lead to the failure of detection. Therefore, it is necessary to determine through condition f that the target object is not vacant.

[0272] Condition g is to ensure the stability of the acquisition frame used to identify the target object in the pixel image. When the target scene is in an environment with poor lighting conditions such as at night, the edge of the target object will be blurred due to dim light and other reasons. At this time, the acquisition frame fluctuates greatly, which can easily lead to unstable video acquisition. Therefore, it is necessary to ensure the lighting conditions in the pixel image through condition g.

[0273] Since the video target frame is a rectangular area and the radar target is a point, in order to construct the mapping between the video target point and the radar target point, it is necessary to select a suitable point from the video target rectangular frame; in this embodiment, the center point of the lower edge of the target frame is selected, because the mapping accuracy constructed by the center point of the lower edge of the target frame is relatively high compared with the center point of the matrix or the pedestrian head point.

[0274] Further, such as Figures 6 to 10 As shown in Figure 1, in order to ensure the accuracy of the data used for model training, the collected data needs to be tested. Figure 6 As shown in the figure, the marked area is where pedestrians appear. If there are only a few tracks in the marked area, the amount of data is not enough to support the training of complex neural networks. Therefore, it is necessary to ensure that the number of tracks in the red area can meet the training requirements.

[0275] The specific detection method is:

[0276] 1) Extract the identification area, such as Figure 7 As shown;

[0277] 2) Perform downsampling, i.e. reduce the image resolution, such as Figure 8 As shown;

[0278] 3) Create a binary image with an initial value of zero;

[0279] 4) If Fig. 9 As shown in the figure, when a pedestrian passes through a certain area in the binary image, the corresponding value is set to 1. At this time, the elements with a value of 1 are unevenly distributed, so the image does not meet the quality requirements;

[0280] 5) If Fig.10 As shown, when the proportion of elements with a value of 1 exceeds 40% and the distribution of elements with a value of 1 is uniform, it can be considered that the data meets the quantity and quality requirements.

[0281] Then, the neural network training is performed based on the aforementioned data. The input to the neural network is 2 nodes, and there is a fully connected layer in the middle with a structure of 2*10*20*20*10*2. The output of the neural network is also 2 nodes.

[0282] There are two neural networks that need to be trained. One neural network is used to input image coordinates and output radar coordinates; the other neural network inputs radar coordinates and outputs pixel coordinates. The ratio of the training set data to the test set data is 4:1, and the total data volume is 2,000,000, which requires approximately 32M storage space.

[0283] The output data is then corrected, wherein the data correction includes deleting redundant data and correcting data; in this embodiment, the process of deleting redundant data is to randomly delete part of the local high-density data to reduce the local density to the average density.

[0284] The data correction process is divided into two sub-processes: 1) If the local error is large, more local data will be added; 2) If the global error is large, some data will be randomly deleted in the global range and new global data will be added, that is, the global data will be updated as a whole.

[0285] The complete process of the above process is as follows Fig.11 As shown,

[0286] Step 1: Initialization. Initialize the network's hyperparameters, including batch size, learning rate, moment value, number of trainings, training set data size, test set data size (corresponding to Fig.11 Step 1) in the above code.

[0287] Step 2: Scene recognition. The purpose of this step is to determine when to collect data. Scene recognition output: whether it is an ideal scene. An ideal scene ensures that the collected data is unambiguous. The scene recognition process runs through the entire algorithm. During the scene recognition process, the radar sensor outputs the number of targets and the corresponding trajectory information. The video outputs the number of targets and the position of the target frame (corresponding to Fig.11 Step 2 in the above example.

[0288] Step 3: Ideal scene judgment (corresponding to Fig.113). If it is an ideal scene, jump to Step 4. Otherwise, jump to Step 2, wait for the next frame of data, and re-perform scene recognition. Based on practice and business needs, most environments are not ideal scenes, but the application scenario allows the algorithm to perform scene recognition for a longer time, that is, the method has enough time to collect data.

[0289] Step 4: Collect data. Although this is an ideal scenario, there is still a lot of work to be done for the data collection business: attach timestamps to radar and video data, select a pixel point in the video target box (corresponding to Fig.11 4 in the previous step).

[0290] Step 5: Time synchronization. Based on the timestamp, paired radar and video data (corresponding to Fig.11 Step 5 in the above example).

[0291] Step 6: Save the data. Save the paired radar and video data to a rewritable storage device. Note that the storage space of the external device must have a certain redundancy (corresponding to Fig.11 6 in the previous step).

[0292] Step 7: Quality and quantity judgment. In order to obtain better training results, it is necessary to judge the quality and quantity of the data set. The quantity is used to ensure that enough data is collected, and the quality is used to ensure that the data is evenly distributed in the fusion area (corresponding to Fig.11 7 in the previous step).

[0293] Step 8: Quality and quantity judgment (corresponding to Fig.11 If satisfied, jump to Step 9; otherwise, jump to Step 12.

[0294] Step 12: Delete redundant data. This situation is usually caused by uneven data. In this case, you can delete the redundant data from the data set (corresponding to Fig.11 12 in the previous step).

[0295] Step 9: Online training. First, divide the collected data into training data set and test data set. Then, import the training data into the neural network and train the network (corresponding to Fig.11 9 in the previous step).

[0296] Step 10: Online testing. After the network training is completed, the obtained network model is tested using the test set data (corresponding to Fig.11 10 in step 10).

[0297] Step 11: By judging (corresponding to Fig.11If the accuracy meets the test requirements, it means that an effective network that can serve radar video sensor fusion is obtained, and jump to Step 14; otherwise, jump to Step 13.

[0298] Step 13: Correct the data (corresponding to Fig.11 This situation is usually caused by insufficient data or incorrect matching. In this case, more data can be added or corrected, and the data can be added and corrected by jumping to Step 2.

[0299] Step 14: Save the network. An effective neural network has been obtained, which will be used for radar video data fusion (corresponding to Fig.11 14 in step 14).

[0300] Step 15: End.

[0301] Among them, the calibrated pixel image is as follows Fig.12 As shown in Figure 2, the calibrated radar is Fig.13 shown.

[0302] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0303] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0304] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0305] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0306] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0307] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0308] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An image calibration method, characterized in that: include: Acquire an initial pixel image and an initial radar image of a target scene, wherein the initial pixel image includes pixel information of a target object contained in the target scene, the initial radar image includes radar information of the target object, and the initial pixel image and the initial radar image are obtained by synchronously shooting the target scene; Performing a first calibration process on the initial pixel image to obtain a target pixel image, wherein the target pixel image is a calibration image of the initial pixel image, and the target pixel image includes the initial pixel image and radar information of the target object; Performing a second calibration process on the initial radar image to obtain a target radar image, wherein the target radar image is a calibrated image of the initial radar image, and the target radar image includes pixel information of the initial radar image and the target object; Performing a fusion operation on the target radar image and the target pixel image to obtain a target data image, wherein the target data image includes the target radar image and the target pixel image; Among them, performing a first calibration process on the initial pixel image includes: identifying the position of the target object and the radar information in the initial pixel image; performing a second calibration process on the initial radar image includes: identifying the position of the target object and the pixel information in the initial radar image.

2. The method according to claim 1, characterized in that: Before acquiring the initial pixel image and the initial radar image of the target scene, the method includes: performing scene recognition on the target scene to determine whether the target scene satisfies a first condition; The acquiring the initial pixel image and the initial radar image of the target scene includes: acquiring the initial pixel image and the initial radar image of the target scene when the target scene satisfies the first condition.

3. The method according to claim 1, characterized in that Performing a first calibration process on the initial pixel image to obtain a target pixel image includes: A first calibration process is performed on the initial pixel image using a first model to obtain a target pixel image, wherein the first model is trained by machine learning using a first data set, the first data set includes multiple groups of first data, each group of the first data includes: a pixel image and a calibration image of the pixel image, the calibration image of the pixel image includes: the pixel image and radar information of the object included in the pixel image.

4. The method according to claim 3, characterized in that Before performing the first calibration process on the initial pixel image, the method further includes: Acquire a first initial data set, wherein the first initial data set includes multiple groups of first data, and each group of first data includes the pixel image and a calibration image of the pixel image; Determining whether the first initial data set satisfies a second condition; When the first initial data set satisfies a second condition, the first initial data set is determined as the first data set.

5. The method according to claim 4, characterized in that Determining whether the first initial data set satisfies the second condition includes: The following operations are performed on each set of data included in the first initial data set to determine a set of data included in the first initial data set whose quality meets a predetermined quality threshold: Extracting a first target area image contained in the pixel image; Performing downsampling processing on the first target area image to obtain a first area image; Binarizing the first region image to obtain a second region image; Performing a binarization result detection on the second region image to obtain a binarization detection result; determining whether the pixel image meets the quality threshold based on the binarization detection result, wherein when the binarization result is a first result, determining that the pixel image meets the quality threshold, and when the binarization result is a second result, determining that the pixel image does not meet the quality threshold; Determine a first number of data groups included in the first initial data set whose quality satisfies the above-mentioned predetermined quality threshold; Based on the first quantity, it is determined whether the first initial data set meets the second condition, wherein when the first quantity is greater than or equal to a quantity threshold, it is determined that the first initial data set meets the second condition, and when the first quantity is less than the quantity threshold, it is determined that the first initial data set does not meet the second condition.

6. The method according to claim 4, characterized in that Determining whether the first initial data set satisfies the second condition includes: Acquire the number of data groups included in the first initial data set, and acquire the quality of each data group included in the first initial data set; Whether the first initial data set satisfies the second condition is determined based on the quantity and the quality, wherein, when it is determined that the quantity satisfies the quantity condition and the quality satisfies the quality condition, it is determined whether the first initial data set satisfies the second condition, and when it is determined that the quantity does not satisfy the quantity condition and / or the quality does not satisfy the quality condition, it is determined that the first initial data set does not satisfy the second condition.

7. The method according to claim 6, characterized in that After determining whether the first initial data set meets the second condition based on the quantity and the quality, the method further includes: In the case where it is determined that the quantity does not satisfy the quantity condition, adding a data group included in the first initial data set; In the case where it is determined that the quality does not satisfy the quality condition, all data groups included in the first data set are deleted, and the data groups included in the first data set are re-determined.

8. The method according to claim 1, characterized in that Performing a second calibration process on the initial radar image to obtain a target radar image includes: The initial radar image is subjected to a second calibration process using a second model to obtain a target radar image, wherein the second model is trained by machine learning using a second data set, the second data set includes multiple groups of second data, each group of the second data includes: a radar image and a calibrated image of the radar image, the calibrated image of the radar image includes: the radar image and pixel information of an object included in the radar image.

9. The method according to claim 8, characterized in that Before performing the second calibration process on the initial radar image, the method further includes: Acquire a second initial data set, wherein the second initial data set includes multiple groups of second data, and each group of second data includes the radar image and a calibration image of the radar image; Determining whether the second initial data set satisfies a second condition; When the second initial data set satisfies a second condition, the second initial data set is determined as the second data set.

10. The method according to claim 9, characterized in that Determining whether the second initial data set satisfies the second condition includes: The following operations are performed for each set of data included in the second initial data set to determine a set of data included in the second initial data set whose quality meets a predetermined quality threshold: Extracting a second target area image contained in the radar image; Downsampling the second target area image to obtain a third area image; performing binarization processing on the third region image to obtain a fourth region image; Performing a binarization result detection on the fourth region image to obtain a binarization detection result; determining whether the radar image meets the quality threshold based on the binarization detection result, wherein if the binarization result is a first result, determining that the radar image meets the quality threshold, and if the binarization result is a second result, determining that the radar image does not meet the quality threshold; Determine a second number of data groups included in the second initial data set whose quality satisfies the above-mentioned predetermined quality threshold; Whether the second initial data set satisfies the second condition is determined based on the second quantity, wherein, when the second quantity is greater than or equal to a quantity threshold, it is determined that the second initial data set satisfies the second condition, and when the second quantity is less than the quantity threshold, it is determined that the second initial data set does not satisfy the second condition.

11. The method according to claim 9, characterized in that Determining whether the second initial data set satisfies the second condition includes: Acquire the number of data groups included in the second initial data set, and acquire the quality of each data group included in the second initial data set; Whether the second initial data set satisfies the second condition is determined based on the quantity and the quality, wherein, when it is determined that the quantity satisfies the quantity condition and the quality satisfies the quality condition, it is determined whether the second initial data set satisfies the second condition, and when it is determined that the quantity does not satisfy the quantity condition and / or the quality does not satisfy the quality condition, it is determined that the second initial data set does not satisfy the second condition.

12. The method according to claim 11, characterized in that After determining whether the second initial data set meets the second condition based on the quantity and the quality, the method further includes: In the case where it is determined that the quantity does not satisfy the quantity condition, adding a data group included in the second initial data set; In the case where it is determined that the quality does not satisfy the quality condition, all data groups included in the second data set are deleted, and the data groups included in the second data set are re-determined.

13. An image calibration method and device, characterized in that: include: An image acquisition module, used to acquire an initial pixel image and an initial radar image of a target scene, wherein the initial pixel image includes pixel information of a target object contained in the target scene, the initial radar image includes radar information of the target object, and the initial pixel image and the initial radar image are obtained by synchronously shooting the target scene; a first calibration module, configured to perform a first calibration process on the initial pixel image to obtain a target pixel image, wherein the target pixel image is a calibration image of the initial pixel image, and the target pixel image includes the initial pixel image and radar information of the target object; a second calibration module, configured to perform a second calibration process on the initial radar image to obtain a target radar image, wherein the target radar image is a calibration image of the initial radar image, and the target radar image includes pixel information of the initial radar image and the target object; a fusion module, configured to perform a fusion operation on the target radar image and the target pixel image to obtain a target data image, wherein the target data image includes the target radar image and the target pixel image; Among them, the first calibration module is used to perform a first calibration processing on the initial pixel image in the following manner: the position of the target object and the radar information are identified in the initial pixel image; the second calibration module is used to perform a second calibration processing on the initial radar image in the following manner: the position of the target object and the pixel information are identified in the initial radar image.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 12 when executed.

15. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 12.

Images