Scene image construction method and device based on multiple sensors and electronic equipment

Abstract

The invention provides a scene image construction method and device based on multiple sensors and electronic equipment. The method comprises the following steps: acquiring original image data of a power scene measured by a plurality of image sensors to obtain a plurality of groups of original image data, and preprocessing the plurality of groups of original image data to obtain a plurality of groups of preprocessed image data; pixel sparse processing is carried out on the preprocessed image data to obtain sparse image data, the pose of the image sensor is preliminarily determined according to the sparse image data, depth information of the image sensor is acquired, and the pose of the image sensor is adjusted according to the depth information; after the pose is adjusted, key frame image data are determined, an electric power scene image is constructed according to the key frame image data, and the key frame image data represent image data with the similarity degree of image contents of two adjacent frame images smaller than a preset threshold value. According to the invention, the problem of tedious steps of establishing the image of the complex power scene in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device, computer-readable storage medium and electronic device for constructing a scene image based on multiple sensors. Background Art

[0002] Existing technical solutions include real-time positioning and mapping methods that use radar point clouds purely, such as ORB-SLAM (a three-dimensional positioning and mapping method based on ORB features), ORB (Oriented FAST and Rotated BRIEF) is an algorithm for image feature extraction and description, and SLAM (Simultaneous Localization and Mapping) is a technology that simultaneously realizes positioning and mapping. ORB-SLAM extracts the ORB features of the point cloud in each time frame, and estimates the depth and optical flow, uses the odometer to estimate the pose at the current time point, and establishes a sparse point cloud map. Even if it is given color information by adding auxiliary color camera back-projection pixels, the map it constructs is also a sparse point cloud map, which only contains rough geometric knowledge and is only helpful for robot positioning, and cannot provide more intuitive and rich color information for other perception tasks. In addition, due to performance optimization, only one local map can be maintained within a period of time, and local loop detection and local BA are required in each local map to refine the pose of all frames in the local area, and the construction of the global map requires multiple local maps, so global loop detection and global BA need to be introduced.

[0003] SLAM technology combined with neural radiation field can achieve dense mapping. It uses neural radiation field as the underlying scene representation method, that is, a 5D function is fitted through a multi-layer perceptron. The input is the coordinates of the world coordinate system and the current camera pose, and the output is the color information and the voxel density of the point. When rendering the image, the pixel value of each pixel is queried by sampling 3D points during the light march to query the color information and voxel density of each point, and then the pixel value is obtained through the volume rendering formula. Even if a dense, geometrically structured, colorful RGB map that can show details can be constructed, it is impossible to render in real time because hundreds or thousands of neural networks need to be queried to obtain the color value of each pixel. And because it is an implicit expression function, it is also cumbersome to convert it into an explicit geometric structure. Summary of the invention

[0004] The main purpose of the present application is to provide a method, device, computer-readable storage medium and electronic device for constructing a scene image based on multiple sensors, so as to at least solve the problem that the image of a complex power scene cannot be established in the prior art.

[0005] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a scene image construction method based on multiple sensors is provided, including: acquiring original image data of an electric power scene measured by multiple image sensors to obtain multiple groups of the original image data, preprocessing the multiple groups of the original image data to obtain multiple groups of preprocessed image data; performing pixel thinning processing on the preprocessed image data to obtain sparse image data, and preliminarily determining the posture of the image sensor based on the sparse image data, acquiring depth information, and determining the final posture of the image sensor based on the depth information; after determining the final posture, determining key frame image data, and constructing an electric power scene image based on the key frame image data, wherein the key frame image data represents image data in which the similarity of the image content of two adjacent frames of images is less than a preset threshold.

[0006] Optionally, multiple groups of the original image data are preprocessed to obtain multiple groups of preprocessed image data, including: performing normal estimation on pixel points in the original image data; performing edge detection on pixel points in the original image data; and performing bilateral filtering on the original image data to obtain the preprocessed image data.

[0007] Optionally, estimating the normal of the pixel points in the original image data includes: using an improved principal component analysis method A=(p j -p mean ) T *W T ·(p j -p mean ) calculates the normal value matrix of the pixel points in the original image data, where A represents the normal value matrix, p j represents the neighborhood point, p mean represents the average value of multiple neighborhood points, and W represents the weight matrix.

[0008] Optionally, edge detection is performed on the pixels in the original image data, including: using the formula The normal vector standard deviation of the neighborhood point is calculated, and when the normal vector standard deviation is greater than the standard deviation threshold, the neighborhood point is determined to be a high curvature candidate point, and the maximum angle difference of the high curvature candidate point is calculated, and when the maximum angle difference is greater than the angle difference threshold, the high curvature candidate point is determined to be a high curvature point, so as to perform edge detection on the original image data, wherein, represents the standard deviation of the normal vector, n i represents the normal vector of the neighborhood point i, n j represents the normal vector of the neighborhood point j, and K1 represents the number of neighborhood points.

[0009] Optionally, performing pixel thinning processing on the preprocessed image data to obtain sparse image data, and preliminarily determining the position and posture of the image sensor based on the sparse image data, including: randomly sampling pixel coordinates in the preprocessed image data so that the sampled image resolution is smaller than the image data of the original resolution of the preprocessed image, to obtain the sparse image data; according to the formula Determine the position and posture of the image sensor, where R represents the rotation matrix, T represents the translation matrix, and L track Indicates the path, C m represents the real image, Represents the image corresponding to the current pose.

[0010] Optionally, determining the final position and posture of the image sensor according to the depth information includes: using formula G select = {G i |G i ∈G and abs(D i -d i ≤ε)} to adjust the position and posture of the image sensor, where G select represents the three-dimensional Gaussian point set of the power scene image, G i represents a point in the three-dimensional Gaussian point set, G represents a pixel point of the power scene image preliminarily determined, and D i Represents the depth information of the point cloud, d i represents the three-dimensional Gaussian depth information, and ε represents the depth threshold.

[0011] Optionally, determining the key frame image data includes: calculating the Gaussian function of the current frame image data to obtain a first Gaussian function, and calculating the Gaussian function of the previous key frame image data to obtain a second Gaussian function; when the similarity between the first Gaussian function and the second Gaussian function is less than the preset threshold, determining that the current frame image data is the key frame image data.

[0012] According to another aspect of the present application, a multi-sensor based scene image construction device is provided, including: a preprocessing unit, used to obtain original image data of an electric power scene measured by multiple image sensors to obtain multiple groups of the original image data, and preprocess the multiple groups of the original image data to obtain multiple groups of preprocessed image data; an adjustment unit, used to perform pixel thinning processing on the preprocessed image data to obtain sparse image data, and preliminarily determine the posture of the image sensor based on the sparse image data, obtain depth information, and determine the final posture of the image sensor based on the depth information; a construction unit, used to determine key frame image data after determining the final posture, and construct an electric power scene image based on the key frame image data, wherein the key frame image data represents image data in which the similarity of image content of two adjacent frames of images is less than a preset threshold.

[0013] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the multi-sensor based scene image construction methods.

[0014] According to another aspect of the present application, an electronic device is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include means for executing any one of the multi-sensor based scene image construction methods.

[0015] Applying the technical solution of the present application, the original image data of the power scene measured by multiple image sensors is obtained to obtain multiple groups of original image data, and the multiple groups of original image data are preprocessed to obtain multiple groups of preprocessed image data; pixel sparse processing is performed on the preprocessed image data to obtain sparse image data, and the posture of the image sensor is preliminarily determined according to the sparse image data, the depth information of the image sensor is obtained, and the posture of the image sensor is adjusted according to the depth information; after adjusting the posture, the key frame image data is determined, and the power scene image is constructed according to the key frame image data, wherein the key frame image data represents the image data whose image content of two adjacent frames is less than a preset threshold. Compared with the cumbersome and complicated steps of constructing a complex power scene map in the prior art, the present application constructs a global map through the above steps, without first constructing a local map, directly performs two-stage posture estimation to obtain a refined posture, obtains a scene image under a complex scene, and realizes real-time rendering. Therefore, it can solve the problem of cumbersome steps in establishing images of complex power scenes in the prior art, and achieve the purpose of simplifying the image construction process and enhancing the image imaging effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings constituting part of the present application are used to provide a further understanding of the present application. The exemplary embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1 A hardware structure block diagram of a mobile terminal for executing a multi-sensor based scene image construction method provided in an embodiment of the present application is shown;

[0018] Figure 2 A schematic diagram of a process of constructing a scene image based on multiple sensors provided in an embodiment of the present application is shown;

[0019] Figure 3 A schematic flow chart of a specific multi-sensor based scene image construction method provided in an embodiment of the present application is shown;

[0020] Figure 4 A structural block diagram of a scene image construction device based on multiple sensors provided in an embodiment of the present application is shown.

[0021] The above drawings include the following reference numerals:

[0022] 102, processor; 104, memory; 106, transmission device; 108, input and output devices. DETAILED DESCRIPTION

[0023] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0024] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] As introduced in the background technology, in the prior art, when establishing images of complex power scenes, the steps are cumbersome, and the global map needs to be composed of multiple local maps. In order to solve the problem of cumbersome steps in establishing images of complex power scenes, the embodiments of the present application provide a scene image construction method, device, computer-readable storage medium and electronic device based on multiple sensors.

[0027] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0028] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 1 is a hardware structure block diagram of a mobile terminal of a scene image construction method based on multiple sensors according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0029] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the scene image construction method based on multiple sensors in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above method is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The transmission device 106 is used to receive or send data via a network. The above-mentioned specific examples of the network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0030] In this embodiment, a multi-sensor based scene image construction method running on a mobile terminal, a computer terminal or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0031] Figure 2 is a flow chart of a method for constructing a scene image based on multiple sensors according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0032] Step S201, acquiring original image data of a power scene measured by multiple image sensors to obtain multiple groups of the original image data, and preprocessing the multiple groups of the original image data to obtain multiple groups of preprocessed image data;

[0033] Specifically, this step aims to reduce the noise of unstructured lidar point cloud data without damaging key features. The original image data is the collected noisy point cloud data. The preprocessing can include normal estimation, edge detection and filtering algorithms. Finally, a clean point cloud that only retains valid information is obtained, that is, the preprocessed image data.

[0034] Step S202, performing pixel thinning processing on the preprocessed image data to obtain sparse image data, and preliminarily determining the pose of the image sensor according to the sparse image data, acquiring depth information, and determining the final pose of the image sensor according to the depth information;

[0035] Specifically, the image sensor is generally a camera. In order to make the image data collected by the image sensor more accurate, the image sensor's pose needs to be corrected or adjusted, which requires first estimating the image sensor's pose. In order to prevent certain pixels representing artifacts in the image from drifting during the above pose optimization process, a coarse-to-fine approach is adopted to optimize the estimated pose in two steps. First, we use the regularity of the image to render only a small number of sparse pixels and optimize the tracking loss to obtain a preliminary, rough camera pose. This approach greatly reduces the impact of artifacts and motion blur. Afterwards, we use the estimated rough camera pose and the observed depth information (radar point cloud) to select a reliable three-dimensional Gaussian that can express clear geometric structure information to represent the final pose of the image sensor.

[0036] Step S203, after determining the final posture, determine key frame image data, and construct a power scene image based on the key frame image data, wherein the key frame image data represents image data in which the similarity of image content between two adjacent frames of images is less than a preset threshold.

[0037] Specifically, after determining the final pose, based on the above-mentioned three-dimensional Gaussian scene representation method, the current frame pose of the image sensor is used as input, and an optimization window is composed of carefully selected key frame image data. Only the key frame image data is used to optimize the Gaussian representation. The key frame image data in the optimization window is registered based on the common visibility between frames, and finally the power scene image is constructed.

[0038] Through this embodiment, the original image data of the power scene measured by multiple image sensors are obtained to obtain multiple groups of original image data, and the multiple groups of original image data are preprocessed to obtain multiple groups of preprocessed image data; pixel sparse processing is performed on the preprocessed image data to obtain sparse image data, and the posture of the image sensor is preliminarily determined according to the sparse image data, the depth information of the image sensor is obtained, and the posture of the image sensor is adjusted according to the depth information; after the posture is adjusted, the key frame image data is determined, and the power scene image is constructed according to the key frame image data, wherein the key frame image data represents the image data of which the similarity of the image content of two adjacent frames of images is less than a preset threshold. Compared with the prior art, the steps of constructing a complex power scene map are cumbersome and complicated. The present application constructs a global map through the above steps, without first constructing a local map, and directly performs two-stage posture estimation to obtain a refined posture, obtains a scene image under a complex scene, and realizes real-time rendering. Therefore, it can solve the problem of cumbersome steps in establishing an image of a complex power scene in the prior art, and achieve the purpose of simplifying the image construction process and enhancing the image imaging effect.

[0039] In the specific implementation process, the above step S201 preprocesses multiple groups of the original image data to obtain multiple groups of preprocessed image data, which can be achieved by the following steps: step S2011: normal estimation of the pixels in the original image data; step S2012: edge detection of the pixels in the original image data; step S2013: bilateral filtering of the original image data to obtain the preprocessed image data. The method performs image preprocessing through the above steps, so that the original data can be uniformly processed to facilitate the construction of subsequent images.

[0040] Specifically, accurate normal estimation is used to determine the local curvature of the underlying geometry and the direction of the points. This method uses an improved principal component analysis method in an iterative process to solve the problem that the traditional principal component analysis method cannot provide accurate normal estimation in the presence of outliers and noise. The improved principal component analysis method uses the projection length as a weight to mitigate the impact of noise and outliers, aiming to achieve a relatively accurate normal estimation, and then each point cloud is iteratively moved along the estimated normal direction.

[0041] Edge detection is usually an important algorithm in point cloud processing. Although normal estimation uses a highly robust iterative method, the position of the point will directly affect the quality of subsequent iterative normal estimation due to large errors in high curvature or low density areas. Therefore, it is necessary to use adaptive strategies for specific points in these special areas. This method adds a maximum angle difference method after curvature detection to enhance the detection of high curvature points and the sharpness of the global structure, solves the problem of difficulty in extracting geometry-related features such as contours, edges and curvature in unstructured point clouds, and realizes the correct detection of special points in special areas of point clouds that require adaptive strategies.

[0042] Filtering algorithms are usually sensitive to pre-set parameters and rely on a large amount of training data. This method alleviates the above dependence by attenuating the displacement of high curvature points in a single iteration, and completes denoising and sharpening based on normal estimation and edge detection and the introduction of bilateral filtering.

[0043] Specifically, bilateral filtering assigns weights based on the distance to neighboring points and the similarity of the normal vector:

[0044]

[0045]

[0046]

[0047] in, represents the standard deviation of the normal vector, n i Represents the neighborhood point p i The normal vector, n j Represents the neighborhood point p j The normal vector of the node, K1 represents the number of neighboring points, ω s ,ω d are the distance weight and normal similarity weight of bilateral filtering, x and y are the distance value and normal similarity value, σ s , σ d They are the distance standard deviation and normal similarity standard deviation of bilateral filtering, respectively. By designing a Gaussian scaling factor s that takes curvature into account to relocate the plane points before using traditional bilateral filtering, the high curvature points can be adjusted more accurately, the angles can be restored more effectively, and the peaks can be ensured to maintain their shape without over-sharpening or smoothing.

[0048] s=exp[-norm(σ) 2 / 2σ 2 ]

[0049]

[0050] where σ is all σ i The standard deviation of Represents the neighborhood point p j Reposition the subsequent points according to the scaling factor s.

[0051] In some optional implementations, step S2011 estimates the normals of the pixels in the original image data, which can be achieved by the following steps: using an improved principal component analysis method A=(p j -p mean ) T *W T ·(p j -p mean ) calculates the normal value matrix of the pixel points in the original image data, where A represents the normal value matrix, p j represents the neighborhood point, p mean represents the average value of the plurality of neighborhood points, and W represents the weight matrix. The method estimates the normal line of the pixel point through the above steps, so that outliers and noise can be removed.

[0052] In the specific implementation process, first, for each point, the KNN algorithm is used to find the k1 nearest neighbors p i , loop through each p i And initialize a weight vector W with all 1s. Then perform iterative operations. When the difference between the normal vector and the previous iteration is less than the proposed threshold, the iteration is considered to have converged. In the iterative process, the average value p of the neighborhood points is first calculated. mean , and then construct the geometric feature matrix, such as the above formula A=(p j -p mean ) T *W T ·(p j -p mean ) as shown.

[0053] Furthermore, the normal vector can be obtained by performing singular value decomposition on A, and then the distance from each neighborhood point to the plane is calculated and normalized to a Gaussian weight, the weight W is updated to the Gaussian weight and the discrete points are removed, and after determining whether convergence, the final estimated normal vector is stored in the normal array. Through the above iterative optimization, the normal estimation value of each point can be finally obtained and the discrete noise points can be periodically removed.

[0054] In some other optional implementations, step S2012 performs edge detection on the pixels in the original image data, which can be achieved by the following steps: The normal vector standard deviation of the neighborhood point is calculated, and when the normal vector standard deviation is greater than the standard deviation threshold, the neighborhood point is determined to be a high curvature candidate point, and the maximum angle difference of the high curvature candidate point is calculated, and when the maximum angle difference is greater than the angle difference threshold, the high curvature candidate point is determined to be a high curvature point, so as to perform edge detection on the original image data, wherein, represents the standard deviation of the normal vector, n i represents the normal vector of the neighborhood point i, n j represents the normal vector of the neighborhood point j, and K1 represents the number of neighborhood points. The method performs edge detection through the above steps, so that pixels with large errors can be detected, making the final image more accurate.

[0055] Specifically, we first calculate the standard deviation of the normal vector in the local neighborhood as above As shown in the formula, after selecting high curvature candidate points using the standard deviation threshold method, the maximum angle difference method is used to verify its accuracy.

[0056] For a high curvature candidate point p i , first query K2 nearest neighbors (K2>K1) and find two adjacent points p i1 and p i2 Its normals are respectively i and n i1 There is a maximum angle difference if arccos(n i ,n i1 ) is large enough, the candidate point is retained as an accurate high curvature point, otherwise it is removed, and the above process iteratively processes all high curvature candidate points.

[0057] In some other optional implementations, the above-mentioned step S202 performs pixel thinning processing on the preprocessed image data to obtain sparse image data, and preliminarily determines the position and posture of the image sensor based on the sparse image data, which can be achieved by the following steps: step S2021: randomly sampling the pixel coordinates in the preprocessed image data so that the sampled image resolution is smaller than the image data of the original resolution of the preprocessed image, to obtain the sparse image data; step S2022: according to the formula Determine the position and posture of the image sensor, where R represents the rotation matrix, T represents the translation matrix, and L track Indicates the path, C m represents the real image, Represents the image corresponding to the current pose. This method determines the preliminary pose through the above steps, so that a rough estimate can reduce the impact of artifacts and motion blur.

[0058] In the specific implementation process, after a new frame is input, the camera pose estimation initializes the pose at the new time step based on the speed consistency assumption of the camera motion, as shown in the following formula.

[0059] E t+1 =E t +(E t -E t-1 )

[0060] Among them, E t is the camera pose at the current time step t, E t+1 is the camera pose at time step t+1, E t-1 is the camera pose at time step t-1.

[0061] The camera pose can then be estimated by rendering the existing scene Gaussian representation image and comparing it with the camera image, and iteratively updating the gradient-based optimization to minimize the rendering color loss, as shown in the formula shown.

[0062] In order to avoid the drift of the optimized pose in the above process caused by some pixels representing artifacts in the image, the estimated pose is optimized in two steps in a coarse-to-fine manner. First, we use the regularity of the image to render only a small number of sparse pixels and optimize the tracking loss to obtain a preliminary, rough camera pose. This method greatly reduces the impact of artifacts and motion blur. Then, we use the estimated rough camera pose and the observed depth information (radar point cloud) to select reliable 3D Gaussians that can express clear geometric structure information.

[0063] In some optional implementations, the above step S202 determines the final position and posture of the image sensor according to the depth information, which can be achieved by the following steps: select = {G i |G i ∈G and abs(D i -d i ≤ε)} to adjust the posture of the image sensor, where G select represents the three-dimensional Gaussian point set of the power scene image, G i represents a point in the three-dimensional Gaussian point set, G represents a pixel point of the power scene image preliminarily determined, and D i Represents the depth information of the point cloud, d i represents the three-dimensional Gaussian depth information, and ε represents the depth threshold.

[0064] In the specific implementation process, in the rough estimation stage, the image information with only 1 / 2 original resolution is rendered by randomly sampling pixel coordinates for optimization; in the refined estimation stage, all visible Gaussian points in the rough pose of the preliminary estimation are queried and Gaussian points outside the scene surface are removed according to the point cloud data provided by the radar. Finally, a reliable three-dimensional Gaussian point set G is used. select To render the full-resolution image and compare it with the input frame, perform pose optimization. In addition, key frames are added based on the ratio of the reliable area of ​​the current observation angle (i.e., the area represented by Gaussian points) to the input frame image and the set threshold.

[0065] In some optional implementations, the above step S203 determines the key frame image data, which can be implemented by the following steps: Step S2031: Calculate the Gaussian function of the current frame image data to obtain the first Gaussian function, and calculate the Gaussian function of the previous key frame image data to obtain the second Gaussian function; Step S2032: When the similarity between the first Gaussian function and the second Gaussian function is less than the preset threshold, determine that the current frame image data is the key frame image data. The method screens out the key frame image data that meets the above conditions through the above steps, so that the power scene image can be constructed more accurately and efficiently.

[0066] Specifically, since the three-dimensional Gaussian representation respects the depth information sorting during the rendering process, common visibility is the evaluation criterion for measuring the relationship between two frames. In a specific camera pose, if a Gaussian point is within the current viewing cone and the related cumulative light transmittance is less than 0.5, then the Gaussian point is considered to be visible in the current view. This method enables the common visibility measurement to determine whether there is occlusion.

[0067] Specifically, co-visibility is measured by measuring the intersection of the Gaussian functions observed between the current frame i and the last keyframe j before it. If the co-visibility is lower than a threshold, or the relative translation of the camera poses of the two is large relative to the intermediate depth, the frame is selected as a keyframe. For more efficient management, following the DSO keyframe management heuristic, only a small number of keyframes are maintained in the current window. If the overlap coefficient with the latest keyframe is lower than the threshold, the keyframe is deleted from the current window. Gaussian function is the primitive for the three-dimensional representation of the scene. Each three-dimensional Gaussian function has a fixed spatial coordinate system xyz after optimization. The Gaussian function that can be observed in the current frame can be obtained by rasterization. Co-visibility is measured by comparing the Gaussian function sets corresponding to the two keyframes. The overlap coefficient can be expressed as the intersection-union ratio.

[0068] The scene representation is optimized in each key frame, and the present invention proposes a coupled Gaussian representation addition and removal method. New Gaussian points will be initialized and inserted into the scene to describe the newly observed area and refine the details in iteration. The relative pose of the point cloud data provided by the radar is projected to the current camera coordinate system to initialize the coordinates of the three-dimensional Gaussian in the world coordinate system. In addition, considering that the point cloud data collected at the current time point is too sparse in some special cases, a new Gaussian is added to the pseudo depth map obtained by rendering at the current pose. For pixels with depth, the new Gaussian is initialized within the range of the depth plus a small variance. For new areas where the depth cannot be rendered, a large variance is added to the average available depth of the rendered pseudo depth map to randomly add new Gaussian points. In fact, most of the Gaussian points added by the above method are redundant or incorrect. Most of the Gaussian points will be automatically removed during optimization. The removal is further accelerated by querying the visibility of the Gaussian in the current key frame window. If the newly added Gaussian points in the last three key frames are not visible by at least three other frames, they will be considered to be geometrically incorrect Gaussian points and forced to be removed.

[0069] After inserting new potential valid Gaussian points, the keyframes in the window are used to reconstruct the currently visible area. In addition, in order to avoid forgetting the previous map, two random previous frames are selected for rendering. In addition, the isotropic regularization term is specially introduced to penalize over-elliptical 3D Gaussians to avoid artifacts caused by extreme elongation in other new perspectives. The overall loss function and the isotropic regularization term are shown in the following formula.

[0070]

[0071] L total =L rgb +λ iso *L iso

[0072] Among them, S i , are the scale and scale mean of the three directions of a specific three-dimensional Gaussian, L rgb is the L1 loss between the rendered image and the real image, Num(G) represents the number of Gaussian points, L iso Indicates the sensitivity loss, λ iso Represents the weight of sensitivity loss.

[0073] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the implementation process of the multi-sensor based scene image construction method of the present application will be described in detail below in combination with specific embodiments.

[0074] This embodiment relates to a specific method for constructing scene images based on multiple sensors, such as Figure 3As shown, the following steps are included:

[0075] Step S1: Data preprocessing, which performs denoising, filtering, calibration and other preprocessing on the raw data from different sensors, especially for the point cloud data full of noise collected by LiDAR, to provide guarantee for subsequent tasks such as camera tracking and 3D mapping, and to meet the needs of availability and high quality of multi-sensor data collection;

[0076] Step S2: Camera pose estimation, estimating the motion trajectory of the working robot in continuous time, achieving efficient determination of its own position in an unknown environment space, and providing accurate position information for real-time mapping and path planning;

[0077] Step S3: Real-time mapping to create a substation scene map with high fidelity, rich details, and realistic geometric structure, which is used to ensure good visibility of slender or complex structure equipment and provide high-quality images and location data for subsequent operation processes and path planning tasks;

[0078] Step S4: Continue to process the multi-sensor collected data at each time point according to the above steps to achieve time-continuous real-time robot positioning and incremental map construction.

[0079] The embodiment of the present application also provides a scene image construction device based on multiple sensors. It should be noted that the scene image construction device based on multiple sensors of the embodiment of the present application can be used to execute the scene image construction method based on multiple sensors provided in the embodiment of the present application. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions have been omitted. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0080] The following is an introduction to the multi-sensor based scene image construction device provided in an embodiment of the present application.

[0081] Figure 4 is a schematic diagram of a scene image construction device based on multiple sensors according to an embodiment of the present application. Figure 4 As shown, the device comprises:

[0082] A preprocessing unit 10 is used to obtain original image data of the power scene measured by multiple image sensors to obtain multiple groups of the original image data, and preprocess the multiple groups of the original image data to obtain multiple groups of preprocessed image data;

[0083] Specifically, this step aims to reduce the noise of unstructured lidar point cloud data without damaging key features. The original image data is the collected noisy point cloud data. The preprocessing can include normal estimation, edge detection and filtering algorithms. Finally, a clean point cloud that only retains valid information is obtained, that is, the preprocessed image data.

[0084] An adjustment unit 20 is used to perform pixel thinning processing on the preprocessed image data to obtain sparse image data, and preliminarily determine the pose of the image sensor according to the sparse image data, obtain depth information, and determine the final pose of the image sensor according to the depth information;

[0085] Specifically, the image sensor is generally a camera. In order to make the image data collected by the image sensor more accurate, the image sensor's pose needs to be corrected or adjusted, which requires first estimating the image sensor's pose. In order to prevent certain pixels representing artifacts in the image from drifting during the above pose optimization process, a coarse-to-fine approach is adopted to optimize the estimated pose in two steps. First, we use the regularity of the image to render only a small number of sparse pixels and optimize the tracking loss to obtain a preliminary, rough camera pose. This approach greatly reduces the impact of artifacts and motion blur. Afterwards, we use the estimated rough camera pose and the observed depth information (radar point cloud) to select a reliable three-dimensional Gaussian that can express clear geometric structure information to represent the final pose of the image sensor.

[0086] The construction unit 30 is used to determine the key frame image data after determining the final posture, and construct the power scene image according to the key frame image data, wherein the key frame image data represents image data in which the similarity of the image content of two adjacent frames of images is less than a preset threshold.

[0087] Specifically, after determining the final pose, a representation device based on the above-mentioned three-dimensional Gaussian scene takes the current frame pose of the image sensor as input, and an optimization window is composed of carefully selected key frame image data. Only the key frame image data is used to optimize the Gaussian representation. The key frame image data in the optimization window is registered based on the common visibility between frames, and finally the power scene image is constructed.

[0088] Through this embodiment, the original image data of the power scene measured by multiple image sensors are obtained to obtain multiple groups of original image data, and the multiple groups of original image data are preprocessed to obtain multiple groups of preprocessed image data; pixel sparse processing is performed on the preprocessed image data to obtain sparse image data, and the posture of the image sensor is preliminarily determined according to the sparse image data, the depth information of the image sensor is obtained, and the posture of the image sensor is adjusted according to the depth information; after the posture is adjusted, the key frame image data is determined, and the power scene image is constructed according to the key frame image data, wherein the key frame image data represents the image data of which the similarity of the image content of two adjacent frames of images is less than a preset threshold. Compared with the prior art, the steps of constructing a complex power scene map are cumbersome and complicated. The present application constructs a global map through the above steps, without first constructing a local map, and directly performs two-stage posture estimation to obtain a refined posture, obtains a scene image under a complex scene, and realizes real-time rendering. Therefore, it can solve the problem of cumbersome steps in establishing an image of a complex power scene in the prior art, and achieve the purpose of simplifying the image construction process and enhancing the image imaging effect.

[0089] In the specific implementation process, the preprocessing unit includes an estimation module, a detection module and a filtering module. The estimation module is used to perform normal estimation on the pixel points in the original image data; the detection module is used to perform edge detection on the pixel points in the original image data; the filtering module is used to perform bilateral filtering on the original image data to obtain the preprocessed image data. The device performs image preprocessing through the above steps, so that the original data can be uniformly processed to facilitate the construction of subsequent images.

[0090] Specifically, accurate normal estimation is used to determine the local curvature of the underlying geometry and the direction of the point. This device uses an improved principal component analysis method in an iterative process to solve the problem that traditional principal component analysis devices cannot provide accurate normal estimation in the presence of outliers and noise. The improved principal component analysis method uses the projection length as a weight to reduce the impact of noise and outliers, aiming to achieve relatively accurate normal estimation, and then each point cloud iteratively moves its position along the estimated normal direction.

[0091] Edge detection is usually an important algorithm in point cloud processing. Although normal estimation uses a highly robust iterative device, the position of the point will directly affect the quality of subsequent iterative normal estimation due to large errors in high curvature or low density areas. Therefore, it is necessary to use an adaptive strategy for specific points in these special areas. This device adds a maximum angle difference device after curvature detection to enhance the detection of high curvature points and the sharpness of the global structure, solving the problem of difficulty in extracting geometry-related features such as contours, edges and curvature in unstructured point clouds, and realizing the correct detection of special points in special areas of point clouds that require adaptive strategies.

[0092] Filtering algorithms are usually sensitive to pre-set parameters and rely on a large amount of training data. The present device alleviates the above dependence by attenuating the displacement of high curvature points in a single iteration, and completes denoising and sharpening based on normal estimation and edge detection and the introduction of bilateral filtering.

[0093] Specifically, bilateral filtering assigns weights based on the distance to neighboring points and the similarity of the normal vector:

[0094]

[0095]

[0096]

[0097] in, represents the standard deviation of the normal vector, n i Represents the neighborhood point p i The normal vector, n j Represents the neighborhood point p j The normal vector of the node, K1 represents the number of neighboring points, ω s ,ω d are the distance weight and normal similarity weight of bilateral filtering, x and y are the distance value and normal similarity value, σ s , σ d They are the distance standard deviation and normal similarity standard deviation of bilateral filtering, respectively. By designing a Gaussian scaling factor s that takes curvature into account to relocate the plane points before using traditional bilateral filtering, the high curvature points can be adjusted more accurately, the angles can be restored more effectively, and the peaks can be ensured to maintain their shape without over-sharpening or smoothing.

[0098] s=exp[-norm(σ) 2 / 2σ 2 ]

[0099]

[0100] where σ is all σ i The standard deviation of Represents the neighborhood point p j Reposition the subsequent points according to the scaling factor s.

[0101] In some optional embodiments, the estimation module includes a calculation submodule for using an improved principal component analysis method A=(p j -p mean ) T *W T ·(p j -p mean) calculates the normal value matrix of the pixel points in the original image data, where A represents the normal value matrix, p j represents the neighborhood point, p mean represents the average value of the plurality of neighborhood points, and W represents the weight matrix. The device estimates the normal line of the pixel point through the above steps, so that outliers and noise can be removed.

[0102] In the specific implementation process, first, for each point, the KNN algorithm is used to find the k1 nearest neighbors p i , loop through each p i And initialize a weight vector W with all 1s. Then perform iterative operations. When the difference between the normal vector and the previous iteration is less than the proposed threshold, the iteration is considered to have converged. In the iterative process, the average value p of the neighborhood points is first calculated. mean , and then construct the geometric feature matrix, such as the above formula A=(p j -p mean ) T *W T ·(p j -p mean ) as shown.

[0103] Furthermore, the normal vector can be obtained by performing singular value decomposition on A, and then the distance from each neighborhood point to the plane is calculated and normalized to a Gaussian weight, the weight W is updated to the Gaussian weight and the discrete points are removed, and after determining whether convergence, the final estimated normal vector is stored in the normal array. Through the above iterative optimization, the normal estimation value of each point can be finally obtained and the discrete noise points can be periodically removed.

[0104] In some other optional embodiments, the detection module includes a determination submodule for determining The normal vector standard deviation of the neighborhood point is calculated, and when the normal vector standard deviation is greater than the standard deviation threshold, the neighborhood point is determined to be a high curvature candidate point, and the maximum angle difference of the high curvature candidate point is calculated, and when the maximum angle difference is greater than the angle difference threshold, the high curvature candidate point is determined to be a high curvature point, so as to perform edge detection on the original image data, wherein, represents the standard deviation of the normal vector, n i represents the normal vector of the neighborhood point i, n j represents the normal vector of the neighborhood point j, and K1 represents the number of neighborhood points. The device performs edge detection through the above steps, so that pixel points with large errors can be detected, making the final image more accurate.

[0105] Specifically, we first calculate the standard deviation of the normal vector in the local neighborhood as above As shown in the formula, after the high curvature candidate points are selected using the standard deviation threshold device, the maximum angle difference device is used to verify its accuracy.

[0106] For a high curvature candidate point p i , first query K2 nearest neighbors (K2>K1) and find two adjacent points p i1 and p i2 Its normals are respectively i and n i1 There is a maximum angle difference if acos(n i ,n i1 ) is large enough, the candidate point is retained as an accurate high curvature point, otherwise it is removed, and the above process iteratively processes all high curvature candidate points.

[0107] In some other optional implementations, the adjustment unit includes: step S2021: randomly sampling the pixel coordinates in the preprocessed image data so that the sampled image resolution is smaller than the image data of the original resolution of the preprocessed image, to obtain the sparse image data; step S2022: according to the formula Determine the position and posture of the image sensor, where R represents the rotation matrix, T represents the translation matrix, and L track Indicates the path, C m represents the real image, The device determines the preliminary posture through the above steps, so that the rough estimation can reduce the influence of artifacts and motion blur.

[0108] In the specific implementation process, after a new frame is input, the camera pose estimation initializes the pose at the new time step based on the speed consistency assumption of the camera motion, as shown in the following formula.

[0109] E t+1 =E t +(E t -E t-1 )

[0110] Among them, E t is the camera pose at the current time step t, E t+1 is the camera pose at time step t+1, E t-1 is the camera pose at time step t-1.

[0111] The camera pose can then be estimated by rendering the existing scene Gaussian representation image and comparing it with the camera image, and iteratively updating the gradient-based optimization to minimize the rendering color loss, as shown in the formula shown.

[0112] In order to avoid the drift of the optimized pose in the above process caused by some pixels representing artifacts in the image, the estimated pose is optimized in two steps in a coarse-to-fine manner. First, we use the regularity of the image to render only a small number of sparse pixels and optimize the tracking loss to obtain a preliminary, rough camera pose. This method greatly reduces the impact of artifacts and motion blur. Then, we use the estimated rough camera pose and the observed depth information (radar point cloud) to select reliable 3D Gaussians that can express clear geometric structure information.

[0113] In some optional embodiments, the adjustment unit includes an adjustment module for adjusting the select = {G i |G i ∈G and abs(D i -d i ≤ε)} to adjust the posture of the image sensor, where G select represents the three-dimensional Gaussian point set of the power scene image, G i represents a point in the three-dimensional Gaussian point set, G represents a pixel point of the power scene image preliminarily determined, and D i Represents the depth information of the point cloud, d i represents the three-dimensional Gaussian depth information, and ε represents the depth threshold.

[0114] In the specific implementation process, in the rough estimation stage, the image information with only 1 / 2 original resolution is rendered by randomly sampling pixel coordinates for optimization; in the refined estimation stage, all visible Gaussian points in the rough pose of the preliminary estimation are queried and Gaussian points outside the scene surface are removed according to the point cloud data provided by the radar. Finally, a reliable three-dimensional Gaussian point set G is used. select To render the full-resolution image and compare it with the input frame, perform pose optimization. In addition, key frames are added based on the ratio of the reliable area of ​​the current observation angle (i.e., the area represented by Gaussian points) to the input frame image and the set threshold.

[0115] In some optional embodiments, the construction unit includes a calculation module and a determination module, the calculation module is used to calculate the Gaussian function of the current frame image data to obtain the first Gaussian function, and calculate the Gaussian function of the previous key frame image data to obtain the second Gaussian function; the determination module is used to determine that the current frame image data is the key frame image data when the similarity between the first Gaussian function and the second Gaussian function is less than the preset threshold. The device screens out the key frame image data that meets the above conditions through the above steps, so that the power scene image can be constructed more accurately and efficiently.

[0116] Specifically, since the three-dimensional Gaussian representation respects the depth information sorting during the rendering process, common visibility is the evaluation criterion for measuring the relationship between two frames. In a specific camera pose, if a Gaussian point is within the current viewing cone and the related cumulative light transmittance is less than 0.5, then the Gaussian point is considered to be visible in the current view. This method enables the common visibility measurement to determine whether there is occlusion.

[0117] Specifically, the co-visibility is measured by measuring the intersection of the Gaussian function observed between the current frame i and the last key frame j before it. If the co-visibility is lower than a threshold, or the relative translation of the camera poses of the two is large relative to the intermediate depth, the frame is selected as a key frame. For more efficient management, following the DSO key frame management heuristic, only a small number of key frames are maintained in the current window. If the overlap coefficient with the latest key frame is lower than the threshold, the key frame is deleted from the current window.

[0118] The scene representation is optimized in each key frame, and the present invention proposes a coupled Gaussian representation adding and removing device. New Gaussian points will be initialized and inserted into the scene to describe the newly observed area and refine the details in iteration. The relative pose of the point cloud data provided by the radar is projected to the current camera coordinate system to initialize the coordinates of the three-dimensional Gaussian in the world coordinate system. In addition, considering that the point cloud data collected at the current time point is too sparse in some special cases, a new Gaussian is added to the pseudo depth map obtained by rendering at the current pose. For pixels with depth, the new Gaussian is initialized within the range of the depth plus a small variance. For new areas where the depth cannot be rendered, a large variance is added to the average available depth of the rendered pseudo depth map to randomly add new Gaussian points. In fact, most of the Gaussian points added by the above device are redundant or incorrect. Most of the Gaussian points will be automatically removed during optimization. The removal is further accelerated by querying the visibility of the Gaussian in the current key frame window. If the newly added Gaussian points in the last three key frames are not visible by at least three other frames, they will be considered as geometrically incorrect Gaussian points and forced to be removed.

[0119] After inserting new potential valid Gaussian points, the keyframes in the window are used to reconstruct the currently visible area. In addition, in order to avoid forgetting the previous map, two random previous frames are selected for rendering. In addition, the isotropic regularization term is specially introduced to penalize over-elliptical 3D Gaussians to avoid artifacts caused by extreme elongation in other new perspectives. The overall loss function and the isotropic regularization term are shown in the following formula.

[0120]

[0121] L total =L rgb +λ iso *L iso

[0122] Among them, S i , are the scale and scale mean of the three directions of a specific three-dimensional Gaussian, L rgb is the L1 loss between the rendered image and the real image, Num(G) represents the number of Gaussian points, L iso Indicates the sensitivity loss, λ iso Represents the weight of sensitivity loss.

[0123] The multi-sensor based scene image construction device includes a processor and a memory, wherein the pre-processing unit, the adjustment unit and the construction unit are stored in the memory as program units, and the processor executes the program units stored in the memory to realize the corresponding functions. The modules are all located in the same processor; or, the modules are located in different processors in any combination.

[0124] The processor includes a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and the image of a complex power scene can be established by adjusting the kernel parameters.

[0125] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0126] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is run, the device where the computer-readable storage medium is located is controlled to execute the multi-sensor based scene image construction method.

[0127] Specifically, the scene image construction method based on multiple sensors includes:

[0128] Step S201, acquiring original image data of a power scene measured by multiple image sensors to obtain multiple groups of the original image data, and preprocessing the multiple groups of the original image data to obtain multiple groups of preprocessed image data;

[0129] Specifically, this step aims to reduce the noise of unstructured lidar point cloud data without damaging key features. The original image data is the collected noisy point cloud data. The preprocessing can include normal estimation, edge detection and filtering algorithms. Finally, a clean point cloud that only retains valid information is obtained, that is, the preprocessed image data.

[0130] Step S202, performing pixel thinning processing on the preprocessed image data to obtain sparse image data, and preliminarily determining the pose of the image sensor according to the sparse image data, acquiring depth information, and determining the final pose of the image sensor according to the depth information;

[0131] Specifically, the image sensor is generally a camera. In order to make the image data collected by the image sensor more accurate, the image sensor's pose needs to be corrected or adjusted, which requires first estimating the image sensor's pose. In order to prevent certain pixels representing artifacts in the image from drifting during the above pose optimization process, a coarse-to-fine approach is adopted to optimize the estimated pose in two steps. First, we use the regularity of the image to render only a small number of sparse pixels and optimize the tracking loss to obtain a preliminary, rough camera pose. This approach greatly reduces the impact of artifacts and motion blur. Afterwards, we use the estimated rough camera pose and the observed depth information (radar point cloud) to select a reliable three-dimensional Gaussian that can express clear geometric structure information to represent the final pose of the image sensor.

[0132] Step S203, after determining the final posture, determine key frame image data, and construct a power scene image based on the key frame image data, wherein the key frame image data represents image data in which the similarity of image content between two adjacent frames of images is less than a preset threshold.

[0133] Specifically, after determining the final pose, based on the above-mentioned three-dimensional Gaussian scene representation method, the current frame pose of the image sensor is used as input, and an optimization window is composed of carefully selected key frame image data. Only the key frame image data is used to optimize the Gaussian representation. The key frame image data in the optimization window is registered based on the common visibility between frames, and finally the power scene image is constructed.

[0134] An embodiment of the present invention provides an electronic device, including a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, at least the following steps are implemented:

[0135] Step S201, acquiring original image data of a power scene measured by multiple image sensors to obtain multiple groups of the original image data, and preprocessing the multiple groups of the original image data to obtain multiple groups of preprocessed image data;

[0136] Specifically, this step aims to reduce the noise of unstructured lidar point cloud data without damaging key features. The original image data is the collected noisy point cloud data. The preprocessing can include normal estimation, edge detection and filtering algorithms. Finally, a clean point cloud that only retains valid information is obtained, that is, the preprocessed image data.

[0137] Step S202, performing pixel thinning processing on the preprocessed image data to obtain sparse image data, and preliminarily determining the pose of the image sensor according to the sparse image data, acquiring depth information, and determining the final pose of the image sensor according to the depth information;

[0138] Specifically, the image sensor is generally a camera. In order to make the image data collected by the image sensor more accurate, the image sensor's pose needs to be corrected or adjusted, which requires first estimating the image sensor's pose. In order to prevent certain pixels representing artifacts in the image from drifting during the above pose optimization process, a coarse-to-fine approach is adopted to optimize the estimated pose in two steps. First, we use the regularity of the image to render only a small number of sparse pixels and optimize the tracking loss to obtain a preliminary, rough camera pose. This approach greatly reduces the impact of artifacts and motion blur. Afterwards, we use the estimated rough camera pose and the observed depth information (radar point cloud) to select a reliable three-dimensional Gaussian that can express clear geometric structure information to represent the final pose of the image sensor.

[0139] Step S203, after determining the final posture, determine key frame image data, and construct a power scene image based on the key frame image data, wherein the key frame image data represents image data in which the similarity of image content between two adjacent frames of images is less than a preset threshold.

[0140] Specifically, after determining the final pose, based on the above-mentioned three-dimensional Gaussian scene representation method, the current frame pose of the image sensor is used as input, and an optimization window is composed of carefully selected key frame image data. Only the key frame image data is used to optimize the Gaussian representation. The key frame image data in the optimization window is registered based on the common visibility between frames, and finally the power scene image is constructed.

[0141] The devices in this article can be servers, PCs, PADs, mobile phones, etc.

[0142] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the method described in each embodiment of the present application:

[0143] Step S201, acquiring original image data of a power scene measured by multiple image sensors to obtain multiple groups of the original image data, and preprocessing the multiple groups of the original image data to obtain multiple groups of preprocessed image data;

[0144] Specifically, this step aims to reduce the noise of unstructured lidar point cloud data without damaging key features. The original image data is the collected noisy point cloud data. The preprocessing can include normal estimation, edge detection and filtering algorithms. Finally, a clean point cloud that only retains valid information is obtained, that is, the preprocessed image data.

[0145] Step S202, performing pixel thinning processing on the preprocessed image data to obtain sparse image data, and preliminarily determining the pose of the image sensor according to the sparse image data, acquiring depth information, and determining the final pose of the image sensor according to the depth information;

[0146] Specifically, the image sensor is generally a camera. In order to make the image data collected by the image sensor more accurate, the image sensor's pose needs to be corrected or adjusted, which requires first estimating the image sensor's pose. In order to prevent certain pixels representing artifacts in the image from drifting during the above pose optimization process, a coarse-to-fine approach is adopted to optimize the estimated pose in two steps. First, we use the regularity of the image to render only a small number of sparse pixels and optimize the tracking loss to obtain a preliminary, rough camera pose. This approach greatly reduces the impact of artifacts and motion blur. Afterwards, we use the estimated rough camera pose and the observed depth information (radar point cloud) to select a reliable three-dimensional Gaussian that can express clear geometric structure information to represent the final pose of the image sensor.

[0147] Step S203, after determining the final posture, determine key frame image data, and construct a power scene image based on the key frame image data, wherein the key frame image data represents image data in which the similarity of image content between two adjacent frames of images is less than a preset threshold.

[0148] Specifically, after determining the final pose, based on the above-mentioned three-dimensional Gaussian scene representation method, the current frame pose of the image sensor is used as input, and an optimization window is composed of carefully selected key frame image data. Only the key frame image data is used to optimize the Gaussian representation. The key frame image data in the optimization window is registered based on the common visibility between frames, and finally the power scene image is constructed.

[0149] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0150] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0151] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0152] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0153] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0154] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0155] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0156] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0157] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0158] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0159] 1) In the scene image construction method based on multiple sensors of the present application, the original image data of the power scene measured by multiple image sensors are obtained to obtain multiple groups of original image data, and the multiple groups of original image data are preprocessed to obtain multiple groups of preprocessed image data; pixel sparse processing is performed on the preprocessed image data to obtain sparse image data, and the posture of the image sensor is preliminarily determined according to the sparse image data, the depth information of the image sensor is obtained, and the posture of the image sensor is adjusted according to the depth information; after adjusting the posture, the key frame image data is determined, and the power scene image is constructed according to the key frame image data, wherein the key frame image data represents the image data of which the similarity of the image content of two adjacent frames is less than a preset threshold. Compared with the cumbersome and complicated steps of constructing a complex power scene map in the prior art, the present application constructs a global map through the above steps, without first constructing a local map, directly performs two-stage posture estimation to obtain a refined posture, obtains a scene image under a complex scene, and realizes real-time rendering. Therefore, it can solve the problem of cumbersome steps in establishing images of complex power scenes in the prior art, and achieve the purpose of simplifying the image construction process and enhancing the image imaging effect.

[0160] 2) In the scene image construction device based on multiple sensors of the present application, the original image data of the power scene measured by multiple image sensors are obtained to obtain multiple groups of original image data, and the multiple groups of original image data are preprocessed to obtain multiple groups of preprocessed image data; pixel sparse processing is performed on the preprocessed image data to obtain sparse image data, and the posture of the image sensor is preliminarily determined according to the sparse image data, the depth information of the image sensor is obtained, and the posture of the image sensor is adjusted according to the depth information; after the posture is adjusted, the key frame image data is determined, and the power scene image is constructed according to the key frame image data, wherein the key frame image data represents the image data whose image content of two adjacent frames is less than a preset threshold. Compared with the cumbersome and complicated steps of constructing a complex power scene map in the prior art, the present application constructs a global map through the above steps, without first constructing a local map, directly performs two-stage posture estimation to obtain a refined posture, obtains a scene image under a complex scene, and realizes real-time rendering. Therefore, it can solve the problem of cumbersome steps in establishing images of complex power scenes in the prior art, and achieve the purpose of simplifying the image construction process and enhancing the image imaging effect.

[0161] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for constructing a scene image based on multiple sensors, characterized in that: include: Acquire raw image data of the power scene measured by multiple image sensors to obtain multiple groups of the raw image data, and pre-process the multiple groups of the raw image data to obtain multiple groups of pre-processed image data; Performing pixel thinning processing on the preprocessed image data to obtain sparse image data, preliminarily determining the pose of the image sensor based on the sparse image data, acquiring depth information, and determining the final pose of the image sensor based on the depth information; After determining the final posture, key frame image data is determined, and a power scene image is constructed based on the key frame image data, wherein the key frame image data represents image data in which the similarity of image content between two adjacent frames of images is less than a preset threshold.

2. The method for constructing scene images based on multiple sensors according to claim 1, characterized in that: Preprocessing the multiple sets of original image data to obtain multiple sets of preprocessed image data includes: Performing normal estimation on pixel points in the original image data; Performing edge detection on pixel points in the original image data; Perform bilateral filtering on the original image data to obtain the preprocessed image data.

3. The method for constructing scene images based on multiple sensors according to claim 2, characterized in that: Estimating normals of pixels in the original image data includes: Using improved principal component analysis method A=(p j -p mean ) T *W T ·(p j -p mean ) calculates the normal value matrix of the pixel points in the original image data, where A represents the normal value matrix, p j represents the neighborhood points, p mean represents the average value of multiple neighborhood points, and W represents the weight matrix.

4. The method for constructing scene images based on multiple sensors according to claim 2, characterized in that: Performing edge detection on pixel points in the original image data includes: By formula The normal vector standard deviation of the neighborhood point is calculated, and when the normal vector standard deviation is greater than the standard deviation threshold, the neighborhood point is determined to be a high curvature candidate point, and the maximum angle difference of the high curvature candidate point is calculated, and when the maximum angle difference is greater than the angle difference threshold, the high curvature candidate point is determined to be a high curvature point, so as to perform edge detection on the original image data, wherein, represents the standard deviation of the normal vector, n i Represents the neighborhood point p i The normal vector, n j Represents the neighborhood point p j The normal vector of , K1 represents the number of neighborhood points.

5. The method for constructing scene images based on multiple sensors according to claim 1, characterized in that: Performing pixel thinning processing on the preprocessed image data to obtain sparse image data, and preliminarily determining the position and posture of the image sensor according to the sparse image data, including: Randomly sampling pixel coordinates in the preprocessed image data so that the resolution of the sampled image is smaller than the original resolution of the preprocessed image, thereby obtaining the sparse image data; According to the formula Determine the position and posture of the image sensor, where R represents the rotation matrix, T represents the translation matrix, and L track Indicates the path, C m represents the real image, Represents the image corresponding to the current pose.

6. The method for constructing scene images based on multiple sensors according to claim 1, characterized in that: Determining a final position and posture of the image sensor according to the depth information includes: By formula G select = {G i |G i ∈G and abs(D i -d i ≤ε)} to adjust the posture of the image sensor, where G select represents the three-dimensional Gaussian point set of the power scene image, G i represents a point in the three-dimensional Gaussian point set, G represents a pixel point of the power scene image preliminarily determined, and D i Represents the depth information of the point cloud, d i represents the three-dimensional Gaussian depth information, and ε represents the depth threshold.

7. The method for constructing scene images based on multiple sensors according to claim 1, characterized in that: Determine the key frame image data, including: Calculate the Gaussian function of the current frame image data to obtain a first Gaussian function, and calculate the Gaussian function of the previous key frame image data to obtain a second Gaussian function; When the similarity between the first Gaussian function and the second Gaussian function is less than the preset threshold, the current frame image data is determined to be the key frame image data.

8. A scene image construction device based on multiple sensors, characterized in that: include: A preprocessing unit, used to acquire original image data of the power scene measured by multiple image sensors to obtain multiple groups of the original image data, and preprocess the multiple groups of the original image data to obtain multiple groups of preprocessed image data; an adjustment unit, configured to perform pixel thinning processing on the preprocessed image data to obtain sparse image data, preliminarily determine the pose of the image sensor according to the sparse image data, obtain depth information, and determine the final pose of the image sensor according to the depth information; A construction unit is used to determine key frame image data after determining the final posture, and construct an electric power scene image based on the key frame image data, wherein the key frame image data represents image data in which the similarity of image content between two adjacent frames of images is less than a preset threshold.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the multi-sensor based scene image construction method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing the multi-sensor based scene image construction method described in any one of claims 1 to 7.

Images