Scene data fusion method and device, equipment and storage medium

Abstract

The invention discloses a scene data fusion method, apparatus and device, and a storage medium, and relates to the field of remote sensing data processing, the method realizes three-party direct fusion of images, point clouds and radars, and compared with other pairwise fusion technologies, the calculation steps of direct superposition are fewer, the error introduction in the intermediate process is reduced, the fusion precision is improved, and the fusion efficiency is improved. The final fusion effect is good, the algorithms are all adaptive algorithms, an absolute static threshold value does not need to be set, the situation that the error of the final fusion result is large due to the fact that the threshold value is set incorrectly is avoided, meanwhile, the weight value of each pixel point fused finally is iterated through global optimization, the optimal fusion effect of the final fusion image is kept, and the fusion efficiency is improved. The problems of image distortion, pixel point missing, tearing and the like caused by direct superposition of a pairwise fusion technology are avoided.

Description

Technical Field

[0001] The present application relates to the technical field of remote sensing data processing, and in particular to a scene data fusion method, device, equipment and storage medium. Background Art

[0002] With the development of remote sensing technology, the ways of acquiring scene data are increasing. Mainstream acquisition methods such as cameras, radars, and three-dimensional mapping have also become important means of perceiving the surrounding environment and have become the mainstream choice recognized by the industry.

[0003] Among them, camera has low cost and low power consumption, and can perform high-level target detection and image processing functions, but is greatly affected by factors such as lighting, weather and image quality; radar has strong adaptability to the environment and can work normally under adverse weather and lighting conditions, while having stable performance and low power consumption, but the spatial resolution is relatively low; 3D mapping has the advantages of high flexibility, variability in data density, rich additional information, and retention of original data, and is widely used, but it also brings the disadvantages of high data processing complexity, large data volume, and non-constructed characteristics, which makes the acquisition of scene data for the same scene have its own advantages and disadvantages.

[0004] Fusing the scene data obtained by the above means is an effective means to improve data accuracy, but the existing fusion methods are usually camera and radar fusion and point cloud and radar fusion, and each fusion method is numerous. Different fusion methods produce different errors and require multiple stages of alignment. If three-party fusion is performed, there are more processing steps, resulting in the continuous accumulation and amplification of errors in each stage, resulting in the final fusion effect with low accuracy. Summary of the invention

[0005] The main purpose of the present application is to provide a scene data fusion method, device, equipment and storage medium to solve the problem that there are many fusion means in the prior art, different fusion means produce different errors, and multiple stages of alignment are required. If three-party fusion is performed, there are many processing steps, resulting in continuous accumulation and amplification of errors in each stage, resulting in low accuracy of the final fusion effect.

[0006] To achieve the above objectives, this application provides the following technical solutions: A method for fusing scene data, wherein the scene data is derived from the same preset scene, and the preset scene has a preset number of calibration points, and the fusing method comprises: Step S1, obtaining the original image data of all calibration points of the preset scene; Step S2, marking all calibration points in the original image data with detection frames using a target detection algorithm to form first-order image data; Step S3, obtaining a point cloud data set having all calibration points of the preset scene through a first preset surveying and mapping strategy; Step S4, projecting all points in the point cloud data set to the first-order image data to form second-order image data; Step S5, obtaining the millimeter radar data of all calibration points of the preset scene through a second preset surveying and mapping strategy; Step S6, mapping the millimeter radar data to the second-order image data through a homography matrix to form third-order image data; Step S7, fusing each pixel of the third-order image data by weighted average method using a weighted transformation method, and obtaining fused image data after traversing all the pixel points; Step S8, iterating the weight value of the weighted average method through a global optimization algorithm so that the clarity evaluation index of the fused image data reaches a maximum value; Step S9, obtaining fused image data corresponding to the maximum value and defining it as final image data.

[0007] As a further improvement of the present application, step S2, marking all calibration points in the original image data with a detection frame by using a target detection algorithm to form first-order image data, includes: Step S21, dividing the original image data into a plurality of square grids on average; Step S22, predicting a number of bounding boxes for all calibrated points based on all square grids according to the target detection algorithm; Step S23, respectively obtaining the confidence of each bounding box, obtaining the bounding box with the largest confidence and marking it as a first-order bounding box; Step S24, calculating the intersection-over-union ratio of each other bounding box with the first-order bounding box; Step S25, selecting a bounding box whose intersection-over-union ratio is greater than or equal to a preset threshold as a second-order bounding box; Step S26, obtaining the second-order bounding box with the highest confidence and defining it as the detection box of all calibration points; Step S27 , outputting all detection frames to the original image data to form the first-order image data.

[0008] As a further improvement of the present application, step S4, projecting all points in the point cloud data set to the first-order image data to form second-order image data, includes: Step S41, obtaining the first The coordinates of the points ; Step S42, defining the plane equation of the first-order image data as ; Step S43, solve the first The projected coordinates of the points : (1); Step S44, solving the projection coordinate values ​​of all points in the point cloud data set according to formula (1); Step S45 , outputting all projection coordinate values ​​to the first-order image data to form the second-order image data.

[0009] As a further improvement of the present application, step S6, mapping the millimeter radar data to the second-order image data through a homography matrix to form third-order image data, includes: Step S61, defining the mapping relationship between the millimeter radar data and the first-order image data by equation (2): (2); in, is the radar position coordinate of the current calibration point, is the pixel coordinate value of the current calibration point based on the original image data, is the homography matrix; Step S62, solving the homography matrix according to equation (2): Obtaining the mapping relationship; Step S63, substituting all radar position coordinates in the millimeter radar data into equation (2) to obtain all radar mapping coordinates; Step S64: output all radar mapping coordinates to the second-order image data to form the third-order image data.

[0010] As a further improvement of the present application, step S7, by using a weighted transformation method, each pixel point of the third-order image data is fused by a weighted average method, and after traversing all the pixel points, the fused image data is obtained, including: Step S71, according to formula (3), each pixel point of the third-order image data is fused by weighted average method: (3); in, is the fused image data, are the pixel coordinates of the image data of each order, for Based on the weight value of the third-order image data, for Based on the weight value of the second-order image data, for Based on the weight value of the first-order image data, is the original image data; Step S72, traversing all pixel points of the third-order image data according to formula (3) to obtain all fused pixel points; Step S73, integrating all fused pixel points to obtain the fused image data.

[0011] As a further improvement of the present application, step S8, iterating the weight value of the weighted average method through a global optimization algorithm so that the clarity evaluation index of the fused image data reaches a maximum value, includes: Step S81, defining a plurality of random solutions for the weight value of the weighted average method according to formula (4), and defining the optimization result of all random solutions as the clarity evaluation index reaching the maximum value; (4); in, is the set of all random solutions, For each random solution, is the label of the random solution, is the number of all random solutions; is the set of velocities of all random solutions, are the speeds of each random solution respectively; Step S82, initialize the position of each random solution, and update the current position and current speed respectively based on the same random solution according to formula (5): (5); in, For the The random solution is The speed of the step, For the The random solution is The speed inertia of the step, is the inertia coefficient, For the The self-perception representation of a random solution, For the social cognitive representation of a random solution; and are learning factors, for A random number, For the The individual optimal solution obtained by random solutions is For the The global optimal solution obtained by random solutions is For the Step 1 A random solution, For the Step 1 A random solution; Step S83, iterate each random solution according to equation (5) to update each and each ; Step S84, determine each Compared with the previous iteration, whether the difference is less than or equal to the first preset adaptation threshold, if each If the difference between the values ​​in the previous iteration and the values ​​in the previous iteration are all less than or equal to the first preset adaptation threshold, step S85 is executed; Step S85, determine each Compared with the difference of the previous iteration, whether it is less than or equal to the second preset adaptation threshold, if each If the difference between the values ​​in the previous iteration and the values ​​in the previous iteration are all less than or equal to the second preset adaptation threshold, step S86 is executed; Step S86, determining that the optimal solution of the weight value has been obtained.

[0012] As a further improvement of the present application, in step S83, each random solution is iterated according to the formula (5) to update each and each ,include: Step S831, based on each iteration, the inertia coefficient is linearly reduced once according to formula (6): (6); in, For the The random solution is The inertia coefficient after step optimization, is the initial inertia coefficient, is the current iteration number, is the maximum number of iteration steps.

[0013] To achieve the above objectives, this application also provides the following technical solutions: A scene data fusion device, the scene data fusion device is applied to the above fusion method, the fusion device comprises: An original image data acquisition module, used to acquire original image data of all calibration points of the preset scene; A first-order image data acquisition module, used to mark all calibration points in the original image data with a detection frame by using a target detection algorithm to form first-order image data; A calibration point cloud data set acquisition module, used to acquire a point cloud data set having all calibration points of the preset scene through a first preset surveying and mapping strategy; A second-order image data acquisition module, used for projecting all points in the point cloud data set to the first-order image data to form second-order image data; A millimeter radar data acquisition module, used to acquire the millimeter radar data of all calibration points of the preset scene through a second preset surveying and mapping strategy; A third-order image data acquisition module, used for mapping the millimeter radar data to the second-order image data through a homography matrix to form third-order image data; A fused image data acquisition module is used to fuse each pixel of the third-order image data by weighted average method through weighted transformation method, and obtain fused image data after traversing all pixel points; A weighted average method weight value acquisition module, used for iterating the weight value of the weighted average method through a global optimization algorithm so that the clarity evaluation index of the fused image data reaches a maximum value; The final image data acquisition module is used to acquire the fused image data corresponding to the maximum value and define it as the final image data.

[0014] To achieve the above objectives, this application also provides the following technical solutions: An electronic device includes a processor and a memory coupled to the processor, wherein the memory stores program instructions executable by the processor; when the processor executes the program instructions stored in the memory, the scene data fusion method as described above is implemented.

[0015] To achieve the above objectives, this application also provides the following technical solutions: A storage medium stores program instructions, which can implement the above-mentioned scene data fusion method when executed by a processor.

[0016] The present application obtains the original image data of all calibration points of a preset scene; marks all calibration points in the original image data with a detection box through a target detection algorithm to form first-order image data; obtains a point cloud data set with all calibration points of the preset scene through a first preset surveying and mapping strategy; projects all points in the point cloud data set to the first-order image data to form second-order image data; obtains the millimeter radar data of all calibration points of the preset scene through a second preset surveying and mapping strategy; maps the millimeter radar data to the second-order image data through a homography matrix to form third-order image data; fuses each pixel point of the third-order image data through a weighted transformation method through a weighted average method, and obtains fused image data after traversing all pixel points; iterates the weight value of the weighted average method through a global optimization algorithm to make the clarity evaluation index of the fused image data reach a maximum value; obtains the fused image data corresponding to the maximum value and defines it as the final image data. The present application realizes the direct fusion of images, point clouds and radars. Compared with other pairwise fusion technologies, the direct superposition has fewer calculation steps, reduces the introduction of errors in the intermediate process, improves the fusion accuracy, and makes the final fusion effect better. The algorithms are all adaptive algorithms, and there is no need to set an absolute still threshold, which avoids large errors in the final fusion result due to incorrect threshold setting. At the same time, the present application also iterates the weight values ​​of the final fused pixels through global optimization to ensure that the final fused image maintains the optimal fusion effect, avoiding image distortion, pixel missing, tearing and other problems caused by direct superposition of pairwise fusion technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic diagram of the process steps of an embodiment of the method for fusing scene data of the present application; Figure 2 A schematic diagram of functional modules of an embodiment of a device for fusing scene data of the present application; Figure 3 This is a schematic diagram of the structure of an embodiment of the electronic device of the present application; Figure 4 This is a schematic diagram of the structure of an embodiment of the storage medium of the present application. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0019] The terms "first", "second" and "third" in this application are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first", "second" and "third" can explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. All directional indications in the embodiments of the present application (such as up, down, left, right, front, back...) are only used to explain the relative position relationship, movement, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication also changes accordingly. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.

[0020] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0021] like Figure 1 As shown, this embodiment provides an embodiment of a method for fusing scene data. In this embodiment, the scene data is derived from the same preset scene, such as natural landforms, building areas, municipal related facilities, etc. The preset scene has a preset number of calibration points, and the number of calibration points is generally two to four.

[0022] Specifically, the fusion method includes the following steps: Step S1, obtaining original image data of all calibration points of a preset scene.

[0023] Preferably, the original image data can be directly captured by a camera. In order to ensure the accuracy of subsequent fusion, several pieces of original image data can be captured at the same shooting position.

[0024] Step S2: Mark all calibration points in the original image data with detection boxes using a target detection algorithm to form first-order image data.

[0025] Preferably, the target detection algorithm can be implemented by VJ, HOG, DPMDetector; deep learning Two-stageRCNN, SPPNet, FastRCNN, FasterRCNN; Trick algorithm FPN, CascadeRCNN; deep learning one-stage Yolo, X, SSD, RetinaNet; deep learning Anchor-free CornerNet, CenterNet, FCOS; TransformerDETR, etc.

[0026] Preferably, this embodiment prefers the Yolo algorithm.

[0027] Step S3: obtaining a point cloud data set with all calibration points of a preset scene through a first preset surveying and mapping strategy.

[0028] Preferably, the first preset surveying and mapping strategy for acquiring point cloud data is one of a laser ranging method, a three-dimensional scanner method, and a structured light method.

[0029] Among them, the laser ranging method uses a laser transmitter to emit a laser to the surface of an object. After the laser hits the surface of the object, the photoelectric converter is used to convert the reflected laser signal into an electrical signal, and the three-dimensional coordinates of the object surface are calculated through an algorithm to obtain point cloud data; the three-dimensional scanner method uses a laser or optical method to scan the surface of an object to obtain point cloud data. Its working principle is to convert the information on the surface of the object into point cloud data by controlling the scanning angle and distance of the scanner; the structured light method uses a method of three-dimensional scanning by projecting a grating pattern. It usually uses a projector to project a specific grating pattern, and then captures the reflected image through a camera to obtain the three-dimensional coordinates of the object surface and obtain point cloud data.

[0030] Preferably, the processing flow of point cloud data is generally data acquisition, point cloud preprocessing, feature extraction and matching. Among them, data acquisition: obtaining point cloud data on the surface of the object. Commonly used data acquisition methods include laser scanning, structured light scanning and image acquisition. Point cloud preprocessing: performing noise filtering, outlier removal, data downsampling and data registration on the collected point cloud data. Feature extraction and matching: extracting features from point cloud data and using matching algorithms to align point clouds.

[0031] Step S4, projecting all points in the point cloud data set to the first-order image data to form second-order image data.

[0032] Step S5, obtaining the millimeter radar data of all calibration points of the preset scene through the second preset surveying and mapping strategy.

[0033] Preferably, the second preset surveying and mapping strategy for obtaining millimeter radar data is usually to directly obtain it on an online platform and measure it by oneself. Online platforms include Journal of Radar, National Comprehensive Earth Observation Data Sharing Platform, China Science Data, IEEE Dataport, Zenodo, Kaggle, Papers With Code, Mendeley Data, Google Dataset Search, and Github. Self-measurement can be directly obtained through millimeter-level radar without difficulty.

[0034] Step S6, mapping the millimeter radar data to the second-order image data through the homography matrix to form the third-order image data.

[0035] Step S7, using a weighted transformation method to fuse each pixel of the third-order image data by a weighted average method, and after traversing all the pixels, the fused image data is obtained.

[0036] Step S8, iterating the weight value of the weighted average method through a global optimization algorithm so that the clarity evaluation index of the fused image data reaches a maximum value.

[0037] Preferably, the clarity evaluation index is mainly divided into a subjective evaluation index and an objective evaluation index.

[0038] Among them, subjective evaluation indicators are evaluated by manually observing images. The results are greatly affected by the subjective factors of the observer and are difficult to quantify. Several common subjective evaluation indicators include: ‌ Clarity level evaluation ‌: Observers classify images into different levels according to their clarity, such as very clear, clear, average, blurry, very blurry, etc. ‌

[0039] Blur rating: Observers rate the image based on its blurriness, for example, 0-10, with higher scores indicating clearer images.

[0040] Detail visibility evaluation: Observers evaluate the details in the image based on how visible they are, such as details are clearly visible, details are vaguely visible, details are not visible, etc.

[0041] Among them, objective evaluation indicators are based on the characteristics of the image itself, are not affected by subjective factors, and can be quantitatively analyzed. Several common objective evaluation indicators include: Grayscale variance: reflects the distribution of grayscale values ​​in an image. The larger the grayscale variance, the clearer the image.

[0042] ‌Edge gradient‌: reflects the sharpness of the edges in the image. The larger the edge gradient, the clearer the image.

[0043] Information entropy: reflects the richness of information in an image. The larger the information entropy, the clearer the image. 1.

[0044] Spectral entropy: reflects the richness of frequency information in an image. The larger the spectral entropy, the clearer the image.

[0045] ‌Structural similarity‌: reflects the similarity of structural information in images. The higher the structural similarity, the clearer the image‌.

[0046] In practical applications, combining multiple indicators can provide a more accurate evaluation. For example, MTF (Modulation Transfer Function) and SFR (Spatial Frequency Response) are important indicators for measuring camera resolution, and these indicators can be used to compare and verify the spatial resolution levels of different cameras. In addition, Brenner gradient method, Tenegrad gradient method, Laplace gradient method, variance method and energy gradient method are also common image clarity evaluation methods.

[0047] Preferably, in this embodiment, one or more combinations of objective evaluation indicators may be selected to evaluate the clarity. If multiple combinations are selected, the average scores of the multiple combinations may be used for evaluation.

[0048] Step S9, obtaining the fused image data corresponding to the maximum value and defining it as the final image data.

[0049] Furthermore, in step S2, all calibration points are marked in the original image data by a detection frame through a target detection algorithm to form first-order image data, including: Step S21, dividing the original image data into a plurality of square grids on average.

[0050] Specifically, the size of the original image of the image data may be adjusted to 448×448, and then the adjusted image may be evenly divided into S×S (eg, 7×7) grids, and the size of each grid is 64×64.

[0051] Step S22, predicting several bounding boxes for all calibrated points based on all square grids according to the target detection algorithm.

[0052] Step S23, respectively obtain the confidence of each bounding box, obtain the bounding box with the largest confidence and mark it as the first-order bounding box.

[0053] Preferably, the confidence level can be understood as whether there is a target in the current grid and the accuracy of the detection frame.

[0054] Step S24, calculating the intersection-over-union ratio of each other bounding box with the first-order bounding box.

[0055] Step S25 , selecting a bounding box whose intersection-over-union ratio is greater than or equal to a preset threshold as a second-order bounding box.

[0056] Preferably, the preset threshold can be set to 80%.

[0057] Step S26, obtaining the second-order bounding box with the highest confidence and defining it as the detection box of all calibration points.

[0058] Step S27 , outputting all detection frames to the original image data to form first-order image data.

[0059] Preferably, each grid is used to predict The horizontal coordinate, vertical coordinate, width, height of each detection box, and the confidence of each detection box, that is, each grid needs to predict value.

[0060] Each grid needs to be predicted indivual ;in, is the offset of the center of the detection box relative to the grid, is the ratio of the detection box to the above resized image, is the confidence of the grid, which takes a value of 1 or 0.

[0061] Preferably, the confidence level can be understood as whether there is a target in the current grid and the accuracy of the detection frame.

[0062] For example: suppose there is a target in a resized image, and the width and height of the resized image are but: Divide the image into 7×7 (S×S) grids evenly, and there is a grid located at the center of the target. The coordinates of the grid are , let the coordinates of the center of the target be , the above offset can be calculated according to the following formula : .

[0063] Preferably, in actual detection, if the predicted detection box and the actual bounding box overlap perfectly, the intersection-and-union ratio is 1. In actual application, the value of the first preset threshold can generally be set to 0.5 to determine whether the predicted bounding box is correct, and the accuracy of the bounding box is positively correlated with the intersection-and-union ratio.

[0064] Preferably, the YOLO algorithm also needs to train the detection frame to improve the accuracy of target detection.

[0065] Next, the training model is trained using a preset pedestrian and vehicle training set, and the weights and biases of the training model are iteratively adjusted a first preset number of times using a back propagation algorithm to reduce the value of the loss function of the training model.

[0066] Preferably, the loss function is as follows: .

[0067] in, For the Grid An indicator function of whether a detection box is responsible for the target, with a value of 1 or 0; , , , , Corresponding to indivual Predicted value.

[0068] It can be understood that the loss function includes the coordinate value deviation of the detection box, the confidence deviation, and the prediction probability deviation (or category deviation).

[0069] in, is the detection frame midpoint loss in the coordinate value deviation, is the loss of detection box width and height in coordinate value deviation, is the confidence deviation, is the deviation of the predicted probability (or class deviation).

[0070] in, is the positioning error penalty. Generally, ; That is the S×S grids mentioned above; is the number of bounding boxes; and For the The estimated values ​​of the horizontal and vertical coordinates of the midpoint of the bounding box; and For the An estimate of the width and height of the bounding box; For the The confidence of the bounding box; For the An estimate of the confidence of the bounding box; is the confidence prediction loss, generally ; For the The class probabilities of the bounding boxes; For the An estimate of the class probability of each bounding box; and In correspond .

[0071] It should be noted that since each grid does not necessarily contain a target, if there is no target in the grid, it will lead to The value of is 0, which makes the gradient span in the subsequent back propagation algorithm too large, so we introduce To control the loss of the predicted position of the detection box, and introduce Controls the penalty for non-existent targets within a single grid.

[0072] It should be noted that the meaning of the symbols used in the principle description of the above target detection algorithm is not interchangeable with the meaning of other symbols in the context.

[0073] Further, in step S4, all points in the point cloud data set are projected onto first-order image data to form second-order image data, including: Step S41, obtaining the first The coordinates of the points .

[0074] Step S42, define the plane equation of the first-order image data as .

[0075] Preferably, if the first-order image data is coplanar with the xy plane, then Can be optimized to , in order to reduce the computing burden.

[0076] Step S43, solve the first The projected coordinates of the points : (1).

[0077] Step S44, solving the projection coordinate values ​​of all points in the point cloud data set according to equation (1).

[0078] In step S45, all projection coordinate values ​​are output to the first-order image data to form the second-order image data.

[0079] Furthermore, step S6, mapping the millimeter radar data to the second-order image data through the homography matrix to form the third-order image data, includes: Step S61, defining the mapping relationship between the millimeter radar data and the first-order image data by equation (2): (2).

[0080] in, is the radar position coordinate of the current calibration point, is the pixel coordinate value of the current calibration point based on the original image data, is the homography matrix.

[0081] Step S62, solving the homography matrix according to equation (2): Get the mapping relationship.

[0082] Step S63: Substitute all radar position coordinates in the millimeter radar data into equation (2) to obtain all radar mapping coordinates.

[0083] In step S64, all radar mapping coordinates are output to second-order image data to form third-order image data.

[0084] For example: If the coordinates of the four positions of the corner reflector detected by the millimeter wave radar They are (12,4.9,0), (16.2,-0.9,0), (16.4,3.1,0), (18.2,-3.7,0), respectively. The corresponding pixel coordinates are It is (373.68,575.71,0), (453.22,544.49,0), (264.38,546.6,0), (177,547.74,0). Then the homography matrix is ​​calculated by formula (2): .

[0085] Furthermore, in step S7, each pixel of the third-order image data is fused by weighted average method by weighted transformation method, and fused image data is obtained after traversing all the pixels, including: Step S71: According to formula (3), each pixel of the third-order image data is fused by weighted average method: (3).

[0086] in, is the fused image data, is the pixel coordinate of each order of image data, for Based on the weight value of the third-order image data, for Based on the weight value of the second-order image data, for Based on the weight value of the first-order image data, The original image data.

[0087] Step S72, traverse all pixel points of the third-order image data according to formula (3) to obtain all fused pixel points.

[0088] Step S73, integrating all fused pixel points to obtain fused image data.

[0089] Furthermore, step S8, iterating the weight value of the weighted average method through a global optimization algorithm so that the clarity evaluation index of the fused image data reaches a maximum value, includes: Step S81, define a number of random solutions for the weight value of the weighted average method according to formula (4), and define the optimization result of all random solutions as the clarity evaluation index reaching the maximum value.

[0090] (4).

[0091] in, is the set of all random solutions, For each random solution, is the label of the random solution, is the number of all random solutions; is the set of velocities of all random solutions, are the speeds of each random solution respectively.

[0092] Step S82, initialize the position of each random solution, and update the current position and current speed respectively based on the same random solution according to formula (5): (5).

[0093] in, For the The random solution is The speed of the step, For the The random solution is The speed inertia of the step, is the inertia coefficient, For the The self-perception representation of a random solution, For the social cognitive representation of a random solution; and are learning factors, for A random number, For the The individual optimal solution obtained by random solutions is For the The global optimal solution obtained by random solutions is For the Step 1 A random solution, For the Step 1 A random solution.

[0094] Preferably, The value range is , preferably ; The value range is , preferably .

[0095] Step S83, iterate each random solution according to formula (5) to update each and each .

[0096] Step S84, determine each Compared with the previous iteration, whether the difference is less than or equal to the first preset adaptation threshold, if each If the difference compared with the previous iteration is less than or equal to the first preset adaptation threshold, step S85 is executed.

[0097] Step S85, determine each Compared with the difference of the previous iteration, whether it is less than or equal to the second preset adaptation threshold, if each If the difference compared with the previous iteration is less than or equal to the second preset adaptation threshold, step S86 is executed.

[0098] Step S86, determining that the optimal solution of the weight value has been obtained.

[0099] Preferably, the values ​​of the first preset adaptation threshold and the second preset adaptation threshold need to be adjusted according to the specific problem, and generally need to be adjusted according to the calculation results. If the adaptation threshold is set too small, the algorithm may stop prematurely and fail to obtain the optimal solution; if the adaptation threshold is set too large, the algorithm may be over-iterated, wasting computing resources.

[0100] Preferably, the adaptation threshold may also be evaluated by one of the Griewank function, the Rastrigin function, the Schaffer function, the Ackley function, and the Rosenbrock function.

[0101] Further, in step S83, each random solution is iterated according to equation (5) to update each and each ,include: Step S831, based on each iteration, the inertia coefficient is linearly reduced once according to formula (6): (6).

[0102] in, For the The random solution is The inertia coefficient after step optimization, is the initial inertia coefficient, is the current iteration number, is the maximum number of iteration steps.

[0103] Preferably, the initial inertia coefficient is generally set to 0.5, and the maximum number of iteration steps is generally set according to actual needs, and in this embodiment can be set to 1000 times.

[0104] This embodiment obtains original image data with all calibration points of a preset scene; marks all calibration points in the original image data with detection boxes through a target detection algorithm to form first-order image data; obtains a point cloud data set with all calibration points of the preset scene through a first preset surveying and mapping strategy; projects all points in the point cloud data set to the first-order image data to form second-order image data; obtains millimeter radar data with all calibration points of the preset scene through a second preset surveying and mapping strategy; maps the millimeter radar data to the second-order image data through a homography matrix to form third-order image data; fuses each pixel point of the third-order image data through a weighted transformation method through a weighted average method, and obtains fused image data after traversing all pixel points; iterates the weight value of the weighted average method through a global optimization algorithm to make the clarity evaluation index of the fused image data reach a maximum value; obtains the fused image data corresponding to the maximum value and defines it as the final image data. This embodiment realizes the direct fusion of images, point clouds and radars. Compared with other pairwise fusion technologies, the direct superposition has fewer calculation steps, reduces the introduction of errors in the intermediate process, improves the fusion accuracy, and makes the final fusion effect better. Moreover, the algorithms are all adaptive algorithms, and there is no need to set an absolute still threshold, which avoids large errors in the final fusion result due to incorrect threshold setting. At the same time, this embodiment also iterates the weight values ​​of the final fused pixels through global optimization to ensure that the final fused image maintains the best fusion effect, avoiding image distortion, pixel missing, tearing and other problems caused by direct superposition of pairwise fusion technologies.

[0105] like Figure 2 As shown, this embodiment provides an embodiment of a fusion device for scene data. In this embodiment, the fusion device is applied to the fusion method in the above embodiment.

[0106] Specifically, the fusion device includes an original image data acquisition module 1, a first-order image data acquisition module 2, a calibration point cloud data set acquisition module 3, a second-order image data acquisition module 4, a millimeter radar data acquisition module 5, a third-order image data acquisition module 6, a fused image data acquisition module 7, a weighted average method weight value acquisition module 8, and a final image data acquisition module 9, which are electrically connected in sequence.

[0107] Among them, the original image data acquisition module 1 is used to acquire the original image data of all calibration points of the preset scene; the first-order image data acquisition module 2 is used to mark all calibration points in the original image data through the detection box through the target detection algorithm to form the first-order image data; the calibration point cloud data set acquisition module 3 is used to acquire the point cloud data set of all calibration points of the preset scene through the first preset surveying and mapping strategy; the second-order image data acquisition module 4 is used to project all points in the point cloud data set to the first-order image data to form the second-order image data; the millimeter radar data acquisition module 5 is used to acquire the millimeter radar data set of all calibration points of the preset scene through the second preset surveying and mapping strategy data; the third-order image data acquisition module 6 is used to map the millimeter radar data to the second-order image data through the homography matrix to form the third-order image data; the fused image data acquisition module 7 is used to fuse each pixel point of the third-order image data through the weighted average method through the weighted transformation method, and obtain the fused image data after traversing all the pixels; the weighted average method weight value acquisition module 8 is used to iterate the weight value of the weighted average method through the global optimization algorithm to make the clarity evaluation index of the fused image data reach the maximum value; the final image data acquisition module 9 is used to obtain the fused image data corresponding to the maximum value and define it as the final image data.

[0108] Furthermore, the first-order image data acquisition module 2 specifically includes a first first-order image data acquisition sub-module, a second first-order image data acquisition sub-module, a third first-order image data acquisition sub-module, a fourth first-order image data acquisition sub-module, a fifth first-order image data acquisition sub-module, a sixth first-order image data acquisition sub-module, and a seventh first-order image data acquisition sub-module, which are electrically connected in sequence; the first first-order image data acquisition sub-module is electrically connected to the original image data acquisition module 1, and the seventh first-order image data acquisition sub-module is electrically connected to the calibration point cloud data set acquisition module 3.

[0109] Among them, the first first-order image data acquisition submodule is used to divide the original image data into a number of square grids on average; the second first-order image data acquisition submodule is used to predict a number of bounding boxes for all calibration points based on all square grids according to the target detection algorithm; the third first-order image data acquisition submodule is used to obtain the confidence of each bounding box respectively, and obtain the bounding box with the largest confidence and mark it as the first-order bounding box; the fourth first-order image data acquisition submodule is used to calculate the intersection and union ratio of each other bounding box with the first-order bounding box respectively; the fifth first-order image data acquisition submodule is used to select the bounding box with an intersection and union ratio greater than or equal to a preset threshold as the second-order bounding box; the sixth first-order image data acquisition submodule is used to obtain the second-order bounding box with the highest confidence and define it as the detection box of all calibration points; the seventh first-order image data acquisition submodule is used to output all detection boxes to the original image data to form the first-order image data.

[0110] Furthermore, the second-order image data acquisition module 4 specifically includes a first second-order image data acquisition sub-module, a second second-order image data acquisition sub-module, a third second-order image data acquisition sub-module, a fourth second-order image data acquisition sub-module, and a fifth second-order image data acquisition sub-module, which are electrically connected in sequence; the first second-order image data acquisition sub-module is electrically connected to the calibration point cloud data set acquisition module 3, and the fifth second-order image data acquisition sub-module is electrically connected to the millimeter radar data acquisition module 5.

[0111] Among them, the first and second order image data acquisition submodule is used to obtain the first The coordinates of the points .

[0112] The second second-order image data acquisition submodule is used to define the plane equation of the first-order image data as: .

[0113] The third second-order image data acquisition submodule is used to solve the first The projected coordinates of the points : (1).

[0114] The fourth second-order image data acquisition submodule is used to solve the projection coordinate values ​​of all points in the point cloud data set according to formula (1).

[0115] The fifth second-order image data acquisition submodule is used to output all projection coordinate values ​​to the first-order image data to form second-order image data.

[0116] Furthermore, the third-order image data acquisition module 6 specifically includes a first third-order image data acquisition submodule, a second third-order image data acquisition submodule, a third third-order image data acquisition submodule, and a fourth third-order image data acquisition submodule, which are electrically connected in sequence; the first third-order image data acquisition submodule is electrically connected to the millimeter radar data acquisition module 5, and the fourth third-order image data acquisition submodule is electrically connected to the fused image data acquisition module 7.

[0117] The first third-order image data acquisition submodule is used to define the mapping relationship between millimeter radar data and first-order image data through formula (2): (2).

[0118] in, is the radar position coordinate of the current calibration point, is the pixel coordinate value of the current calibration point based on the original image data, is the homography matrix.

[0119] The second and third order image data acquisition submodule is used to solve the homography matrix according to formula (2): Get the mapping relationship.

[0120] The third third-order image data acquisition submodule is used to substitute all radar position coordinates in the millimeter radar data into equation (2) to obtain all radar mapping coordinates.

[0121] The fourth third-order image data acquisition submodule is used to output all radar mapping coordinates to second-order image data to form third-order image data.

[0122] Furthermore, the fused image data acquisition module 7 specifically includes a first fused image data acquisition submodule, a second fused image data acquisition submodule, and a third fused image data acquisition submodule, which are electrically connected in sequence; the first fused image data acquisition submodule is electrically connected to the fourth third-order image data acquisition submodule, and the third fused image data acquisition submodule is electrically connected to the weighted average method weight value acquisition module 8.

[0123] The first fused image data acquisition submodule is used to fuse each pixel of the third-order image data by weighted average method according to formula (3): (3).

[0124] in, is the fused image data, is the pixel coordinate of each order of image data, for Based on the weight value of the third-order image data, for Based on the weight value of the second-order image data, for Based on the weight value of the first-order image data, The original image data.

[0125] The second fused image data acquisition submodule is used to obtain all fused pixel points by traversing all pixel points of the third-order image data according to formula (3).

[0126] The third fused image data acquisition submodule is used to integrate all fused pixel points to obtain fused image data.

[0127] Furthermore, the weighted average method weight value acquisition module 8 specifically includes a first weighted average method weight value acquisition submodule, a second weighted average method weight value acquisition submodule, a third weighted average method weight value acquisition submodule, a fourth weighted average method weight value acquisition submodule, a fifth weighted average method weight value acquisition submodule, and a sixth weighted average method weight value acquisition submodule, which are electrically connected in sequence; the first weighted average method weight value acquisition submodule is electrically connected to the third post-fusion image data acquisition submodule, and the sixth weighted average method weight value acquisition submodule is electrically connected to the final image data acquisition module 9.

[0128] The first weighted average method weight value acquisition submodule is used to define a number of random solutions for the weight value of the weighted average method according to formula (4), and define the optimization result of all random solutions as the clarity evaluation index reaching the maximum value.

[0129] (4).

[0130] in, is the set of all random solutions, For each random solution, is the label of the random solution, is the number of all random solutions; is the set of velocities of all random solutions, are the speeds of each random solution respectively.

[0131] The second weighted average weight value acquisition submodule is used to initialize the position of each random solution, and based on the same random solution, update the current position and current speed according to formula (5): (5).

[0132] in, For the The random solution is The speed of the step, For the The random solution is The speed inertia of the step, is the inertia coefficient, For the The self-perception representation of a random solution, For the social cognitive representation of a random solution; and are learning factors, for A random number, For the The individual optimal solution obtained by random solutions is For the The global optimal solution obtained by random solutions is For the Step 1 A random solution, For the Step 1 A random solution.

[0133] The third weighted average weight value acquisition submodule is used to iterate each random solution according to formula (5) to update each and each .

[0134] The fourth weighted average method weight value acquisition submodule is used to determine each Whether the difference compared with the previous iteration is less than or equal to the first preset adaptation threshold.

[0135] The fifth weighted average weight value acquisition submodule is used for each Compared with the difference of the previous iteration, if the difference is less than or equal to the first preset adaptation threshold, then each Whether the difference compared with the previous iteration is less than or equal to the second preset adaptation threshold.

[0136] The sixth weighted average weight value acquisition submodule is used for each If the difference values ​​compared to the previous iteration are all less than or equal to the second preset adaptation threshold, it is determined that the optimal solution of the weight value has been obtained.

[0137] Furthermore, the third weighted average weight value acquisition submodule is specifically used to linearly decrease the inertia coefficient once according to formula (6) based on each iteration step: (6).

[0138] in, For the The random solution is The inertia coefficient after step optimization, is the initial inertia coefficient, is the current iteration number, is the maximum number of iteration steps.

[0139] It should be noted that this embodiment is a functional module item embodiment based on the above-mentioned method embodiment. For additional contents such as the preference, expansion, and example illustration of this embodiment, please refer to the above-mentioned method embodiment, and this embodiment will not be repeated here.

[0140] This embodiment obtains original image data with all calibration points of a preset scene; marks all calibration points in the original image data with detection boxes through a target detection algorithm to form first-order image data; obtains a point cloud data set with all calibration points of the preset scene through a first preset surveying and mapping strategy; projects all points in the point cloud data set to the first-order image data to form second-order image data; obtains millimeter radar data with all calibration points of the preset scene through a second preset surveying and mapping strategy; maps the millimeter radar data to the second-order image data through a homography matrix to form third-order image data; fuses each pixel point of the third-order image data through a weighted transformation method through a weighted average method, and obtains fused image data after traversing all pixel points; iterates the weight value of the weighted average method through a global optimization algorithm to make the clarity evaluation index of the fused image data reach a maximum value; obtains the fused image data corresponding to the maximum value and defines it as the final image data. This embodiment realizes the direct fusion of images, point clouds and radars. Compared with other two-by-two fusion technologies, the direct superposition has fewer calculation steps, reduces the introduction of errors in the intermediate process, improves the fusion accuracy, and makes the final fusion effect better. Moreover, the algorithms are all adaptive algorithms, and there is no need to set an absolute still threshold, which avoids large errors in the final fusion result due to incorrect threshold setting. At the same time, this embodiment also iterates the weight values ​​of the final fused pixels through global optimization, so that the final fused image maintains the best fusion effect, avoiding image distortion, pixel missing, tearing and other problems caused by direct superposition of two-by-two fusion technologies.

[0141] Figure 3 An embodiment of the electronic device of the present application is shown, see Figure 3 The electronic device 10 includes a processor 101 and a memory 102 coupled to the processor 101 .

[0142] The memory 102 stores program instructions for implementing the scene data fusion method of any of the above embodiments.

[0143] The processor 101 is used to execute program instructions stored in the memory 102 to fuse scene data.

[0144] The processor 101 may also be referred to as a CPU (Central Processing Unit). The processor 101 may be an integrated circuit chip having signal processing capabilities. The processor 101 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0145] Further, Figure 4 This is a schematic diagram of the structure of a storage medium according to an embodiment of the present application. Figure 4 The storage medium 11 of the embodiment of the present application stores program instructions 111 that can implement all the above methods, wherein the program instructions 111 can be stored in the above storage medium in the form of a software product, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or terminal devices such as a computer, a server, a mobile phone, and a tablet.

[0146] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0147] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units. The above is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the specification and drawings of this application, or directly or indirectly used in other related technical fields, is also included in the patent protection scope of the present application.

[0148] The specific implementation methods of the application are described in detail above, but they are only examples, and the application is not limited to the specific implementation methods described above. For those skilled in the art, any equivalent modification or substitution of the application is also within the scope of the application, and therefore, the equalization, modification, and improvement made without departing from the spirit and principle of the application should be included in the scope of the application.

Claims

1. A method for fusing scene data, wherein the scene data is derived from the same preset scene, and the preset scene has a preset number of calibration points, characterized in that: The fusion method comprises: Step S1, obtaining the original image data of all calibration points of the preset scene; Step S2, marking all calibration points in the original image data with detection frames using a target detection algorithm to form first-order image data; Step S3, obtaining a point cloud data set having all calibration points of the preset scene through a first preset surveying and mapping strategy; Step S4, projecting all points in the point cloud data set to the first-order image data to form second-order image data; Step S5, obtaining the millimeter radar data of all calibration points of the preset scene through a second preset surveying and mapping strategy; Step S6, mapping the millimeter radar data to the second-order image data through a homography matrix to form third-order image data; Step S7, fusing each pixel of the third-order image data by weighted average method using a weighted transformation method, and obtaining fused image data after traversing all the pixel points; Step S8, iterating the weight value of the weighted average method through a global optimization algorithm so that the clarity evaluation index of the fused image data reaches a maximum value; Step S9, obtaining fused image data corresponding to the maximum value and defining it as final image data.

2. The fusion method according to claim 1, characterized in that: Step S2, marking all calibration points in the original image data with a detection frame by using a target detection algorithm to form first-order image data, including: Step S21, dividing the original image data into a plurality of square grids on average; Step S22, predicting a number of bounding boxes for all calibrated points based on all square grids according to the target detection algorithm; Step S23, respectively obtaining the confidence of each bounding box, obtaining the bounding box with the largest confidence and marking it as a first-order bounding box; Step S24, calculating the intersection-over-union ratio of each other bounding box with the first-order bounding box; Step S25, selecting a bounding box whose intersection-over-union ratio is greater than or equal to a preset threshold as a second-order bounding box; Step S26, obtaining the second-order bounding box with the highest confidence and defining it as the detection box of all calibration points; Step S27 , outputting all detection frames to the original image data to form the first-order image data.

3. The fusion method according to claim 1, characterized in that: Step S4, projecting all points in the point cloud data set to the first-order image data to form second-order image data, including: Step S41, obtaining the first The coordinates of the points ; Step S42, defining the plane equation of the first-order image data as ; Step S43, solve the first The projected coordinates of the points : （1）； Step S44, solving the projection coordinate values ​​of all points in the point cloud data set according to formula (1); Step S45 , outputting all projection coordinate values ​​to the first-order image data to form the second-order image data.

4. The fusion method according to claim 1, characterized in that: Step S6, mapping the millimeter radar data to the second-order image data through a homography matrix to form third-order image data, including: Step S61, defining the mapping relationship between the millimeter radar data and the first-order image data by equation (2): （2）； in, is the radar position coordinate of the current calibration point, is the pixel coordinate value of the current calibration point based on the original image data, is the homography matrix; Step S62, solving the homography matrix according to equation (2): Obtaining the mapping relationship; Step S63, substituting all radar position coordinates in the millimeter radar data into equation (2) to obtain all radar mapping coordinates; Step S64: output all radar mapping coordinates to the second-order image data to form the third-order image data.

5. The fusion method according to claim 1, characterized in that: Step S7, fusing each pixel of the third-order image data by weighted average method by weighted transformation method, and obtaining fused image data after traversing all pixel points, including: Step S71, according to formula (3), each pixel point of the third-order image data is fused by weighted average method: （3）； in, is the fused image data, are the pixel coordinates of the image data of each order, for Based on the weight value of the third-order image data, for Based on the weight value of the second-order image data, for Based on the weight value of the first-order image data, is the original image data; Step S72, traversing all pixel points of the third-order image data according to formula (3) to obtain all fused pixel points; Step S73, integrating all fused pixel points to obtain the fused image data.

6. The fusion method according to claim 1, characterized in that: Step S8, iterating the weight value of the weighted average method through a global optimization algorithm so that the clarity evaluation index of the fused image data reaches a maximum value, including: Step S81, defining a plurality of random solutions for the weight value of the weighted average method according to formula (4), and defining the optimization result of all random solutions as the clarity evaluation index reaching the maximum value; （4）； in, is the set of all random solutions, For each random solution, is the label of the random solution, is the number of all random solutions; is the set of velocities of all random solutions, are the speeds of each random solution respectively; Step S82, initialize the position of each random solution, and update the current position and current speed respectively based on the same random solution according to formula (5): （5）； in, For the The random solution is The speed of the step, For the The random solution is The speed inertia of the step, is the inertia coefficient, For the The self-perception representation of a random solution, For the social cognitive representation of a random solution; and are learning factors, for A random number, For the The individual optimal solution obtained by random solutions is For the The global optimal solution obtained by random solutions is For the Step 1 A random solution, For the Step 1 A random solution; Step S83, iterate each random solution according to equation (5) to update each and each ; Step S84, determine each Compared with the previous iteration, whether the difference is less than or equal to the first preset adaptation threshold, if each If the difference between the values ​​in the previous iteration and the values ​​in the previous iteration are all less than or equal to the first preset adaptation threshold, step S85 is executed; Step S85, determine each Compared with the difference of the previous iteration, whether it is less than or equal to the second preset adaptation threshold, if each If the difference between the values ​​in the previous iteration and the values ​​in the previous iteration are all less than or equal to the second preset adaptation threshold, step S86 is executed; Step S86, determining that the optimal solution of the weight value has been obtained.

7. The fusion method according to claim 6, characterized in that: Step S83, iterate each random solution according to equation (5) to update each and each ,include: Step S831, based on each iteration, the inertia coefficient is linearly reduced once according to formula (6): （6）； in, For the The random solution is The inertia coefficient after step optimization, is the initial inertia coefficient, is the current iteration number, is the maximum number of iteration steps.

8. A scene data fusion device, the scene data fusion device is applied to the fusion method according to any one of claims 1 to 7, characterized in that: The fusion device comprises: An original image data acquisition module, used to acquire original image data of all calibration points of the preset scene; A first-order image data acquisition module, used to mark all calibration points in the original image data with a detection frame by using a target detection algorithm to form first-order image data; A calibration point cloud data set acquisition module, used to acquire a point cloud data set having all calibration points of the preset scene through a first preset surveying and mapping strategy; A second-order image data acquisition module, used for projecting all points in the point cloud data set to the first-order image data to form second-order image data; A millimeter radar data acquisition module, used to acquire the millimeter radar data of all calibration points of the preset scene through a second preset surveying and mapping strategy; A third-order image data acquisition module, used for mapping the millimeter radar data to the second-order image data through a homography matrix to form third-order image data; A fused image data acquisition module is used to fuse each pixel of the third-order image data by weighted average method through weighted transformation method, and obtain fused image data after traversing all pixel points; A weighted average method weight value acquisition module, used for iterating the weight value of the weighted average method through a global optimization algorithm so that the clarity evaluation index of the fused image data reaches a maximum value; The final image data acquisition module is used to acquire the fused image data corresponding to the maximum value and define it as the final image data.

9. An electronic device, characterized in that: It comprises a processor and a memory coupled to the processor, wherein the memory stores program instructions executable by the processor; when the processor executes the program instructions stored in the memory, the fusion method as described in any one of claims 1 to 7 is implemented.

10. A storage medium, characterized in that: The storage medium stores program instructions, and when the program instructions are executed by the processor, the fusion method according to any one of claims 1 to 7 can be implemented.

Images