Image processing method and inspection device

Abstract

The present application belongs to the technical field of inspection image processing, and particularly relates to an image processing method and an inspection device, comprising the following steps: obtaining a depth image in front of the inspection device through a depth camera of the inspection device; performing feature object recognition on the depth image to obtain a plurality of feature objects and a distance from each feature object to the depth camera; obtaining an original image of the distance from each feature object to the depth camera by using each wide-angle camera, wherein edges of the original images obtained by the wide-angle cameras at adjacent positions overlap; splicing a plurality of original images of the distance from each feature object to the depth camera to obtain a plurality of panoramic spliced images; and performing element monitoring on each panoramic spliced image to obtain environmental elements in the panoramic spliced image. While ensuring the definition at the positions of the feature objects, image ghosting caused by different focal lengths of multiple images is avoided.

Description

Technical Field

[0001] This invention belongs to the field of inspection image processing technology, specifically relating to an image processing method and inspection equipment. Background Technology

[0002] With the continuous and rapid development of society, economy, and science and technology, the demand for mineral materials is also increasing rapidly. Faced with increasingly information-based mine acquisition and exploration systems, improving overall management level and operational efficiency, and ensuring the safety of miners' lives and property are of paramount importance. Therefore, recording personnel movement, environmental monitoring, and disaster early warning within the mine is crucial. This recording relies heavily on the processing and recognition of captured images by cameras. For vision-based inspection equipment in roadways, the detection of the surrounding environment differs significantly from normal traffic environments. Compared to normal traffic environments, network conditions are poorer, the surrounding environment is smaller, and the number of feature elements is fewer, while the identification of features such as lights, people, pipes, and signs is more precise.

[0003] In existing technologies, wide-angle cameras on inspection equipment can often only capture clear images within a certain distance and a certain angle. At the same time, when multiple wide-angle cameras are shooting simultaneously, their focal lengths may be different. When images with different focal lengths are stitched together in a panoramic image, ghosting will occur. This makes it impossible to accurately judge problems in relevant scenes, and the resulting feature images have low clarity for mine workers, which can easily lead to misjudgments. Summary of the Invention

[0004] In view of the shortcomings of the prior art described above, the purpose of this application is to provide an image processing method for an inspection device including a depth camera and multiple wide-angle cameras, comprising the following steps: The depth image in front of the inspection equipment is obtained through the depth camera of the inspection equipment; The depth image is subjected to feature identification to obtain several features and the distance of each feature to the depth camera; Using each of the wide-angle cameras, a raw image of the distance from each feature to the depth camera is acquired, wherein the edges of the raw images acquired by the wide-angle cameras at adjacent locations overlap; Multiple original images of the distance from each feature to the depth camera are stitched together to obtain several panoramic stitched images; Element monitoring is performed on each of the panoramic stitched images to obtain the environmental elements in the panoramic stitched images.

[0005] In an optional embodiment of this application, in the step of performing element monitoring on each of the panoramic stitched images to obtain environmental elements in the panoramic stitched images, the environmental elements include one or more of the following: intersections, people, vehicles, signs, pipes, and lights.

[0006] In an optional embodiment of this application, the step of identifying features in the depth image to obtain a number of features and the distance of each feature to the depth camera further includes: obtaining inspection equipment location information and obtaining the location of the features based on the inspection equipment location information.

[0007] In an optional embodiment of this application, the step of obtaining the location information of the inspection equipment and obtaining the location of the feature based on the location information of the inspection equipment includes: Obtain the starting position of the inspection equipment and the speed-time data of the inspection equipment from the starting position to the current time; The moving distance of the inspection equipment is obtained based on the speed-time data; The position of the inspection equipment in the inspection route is obtained based on the moving distance and the starting position; The position of each feature is obtained based on the position of the inspection device in the inspection line and the distance of each feature from the depth camera.

[0008] In an optional embodiment of this application, the step of stitching together multiple original images of the distance from each of the features to the depth camera to obtain several panoramic stitched images includes: For each of the original images representing the distance from the feature to the depth camera, obtain matching point pairs between adjacent original images; Remove erroneous matches from matching point pairs to obtain valid matches; Multiple original images of the distance from each feature to the depth camera are stitched together based on valid matching points to obtain several panoramic stitched images.

[0009] In an optional embodiment of this application, the step of obtaining matching point pairs between adjacent original images of multiple original images of the distance from each of the features to the depth camera includes: For each feature and the distance to the depth camera, multiple original images are used to match adjacent original images with wide-angle cameras using the SIFT algorithm to obtain matching point pairs between adjacent original images.

[0010] In an optional embodiment of this application, the step of removing erroneous matching points from matching point pairs to obtain valid matching points includes: The RANSAC algorithm is used to remove mismatched points from the matching point pairs in order to obtain valid matching points.

[0011] In an optional embodiment of this application, the step of stitching together multiple original images of the distance from each of the features to the depth camera based on valid matching points to obtain a plurality of panoramic stitched images includes: The global homography matrix H between two adjacent original images is obtained based on the valid matching points; The global homography matrix H is solved using the DLT algorithm to obtain matrix A; Divide one of the adjacent original images into a grid and obtain the center point of each grid. Calculate the Euclidean distance between each of the grid center points and the valid matching points; Calculate the weight W based on the Euclidean distance; Substitute the weight W into the matrix A to obtain the W*A matrix; The local homography matrix of the current mesh is obtained by decomposing the W*A matrix using the SVD algorithm; Traverse each grid and map all the local homography matrices onto the panoramic canvas to obtain the panoramic stitched image.

[0012] In an optional embodiment of this application, after stitching together multiple original images of the distance from each of the features to the depth camera to obtain several panoramic stitched images, the following steps are further included: The panoramic stitched image is compressed using the Seam Carving algorithm to obtain a compressed image; The Seam Carving algorithm includes the following steps: Calculate the image energy in the panoramic stitched image. The formula is: ; in, This represents the absolute value of the gradient in the x-direction of the panoramic stitched image. Let A and B be the absolute value of the gradient in the y-direction of the panoramic stitched image, where A and B are preset coefficients, A+B=2, and A>B; Calculate the cost map and path map based on image energy; Find and remove the seams with the lowest energy in the panoramic stitched image based on the cost map and path map. Repeat the steps in the Seam Carving algorithm until the preset number of times is reached to obtain the compressed image.

[0013] This application also provides an inspection device, including: Vehicle body; A depth camera, located at the front of the vehicle body; The wide-angle camera includes at least one first wide-angle camera and a second wide-angle camera located on both sides of the vehicle's direction of movement. The first wide-angle camera faces the front or rear of the vehicle, and the second wide-angle camera faces the side of the vehicle. An image processing unit is used to implement the above-mentioned image processing method to identify environmental elements in front of the inspection equipment; The control unit is used to control the movement of the inspection equipment.

[0014] The technical advantages of this invention are as follows: This invention uses a depth camera to acquire the location of a feature in real time, then acquires the corresponding focal length, and a wide-angle camera adjusts its focal length to obtain an original image with clear feature information. In this way, when multiple wide-angle cameras acquire images corresponding to the same feature, they only need to acquire images with the same focal length, which ensures the clarity at the location of the feature and avoids image ghosting caused by different focal lengths in multiple images corresponding to the same feature. Attached Figure Description

[0015] Figure 1 This is a flowchart of the image processing method of the present invention.

[0016] Figure 2 This is a flowchart of the splicing method of the present invention.

[0017] Figure 3 This is a schematic diagram of depth image acquisition by the depth camera in this invention.

[0018] Figure 4 This is a schematic diagram of the wide-angle camera of the present invention acquiring the original image.

[0019] Figure 5 This is a simplified structural diagram of the inspection equipment of the present invention. Detailed Implementation

[0020] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application.

[0021] Please refer to the accompanying drawings. It should be noted that the illustrations provided in this embodiment are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0022] like Figure 1 As shown, an image processing method for an inspection device including a depth camera and multiple wide-angle cameras includes the following steps: S1: Obtain a depth image of the front of the inspection equipment through the depth camera 20 of the inspection equipment.

[0023] The depth camera 20 of the inspection equipment is positioned in front of the equipment, while wide-angle cameras 30 are positioned in front, behind, and to the sides of the equipment. Preferably, four wide-angle cameras 30 are positioned at the front, back, left, and right. During operation, the inspection equipment moves forward, and the depth camera 20 continuously acquires depth images of the area in front of it. The depth camera 20 has high precision, avoiding color distortion caused by changes in lighting conditions. Furthermore, the depth camera 20 can accurately capture the contours, depth, and shape of objects in low-light environments, making it more suitable for working in dimly lit alleyways.

[0024] S2: Perform feature recognition on the depth image to obtain several features and the distance of each feature to the depth camera 20.

[0025] The depth camera 20 acquires several features and the distance of each feature to the depth camera 20 by combining an imaging sensor and light sources such as infrared lasers to measure the time required for light to travel from the camera to the target object and back, thereby obtaining the distance information of the target object. For example... Figure 3 The diagram shown illustrates how depth images are acquired by the depth camera 20. Figure 3 The dashed line represents the distance between the feature object and the wall acquired by the depth camera 20. If no feature object is identified in the depth image, the inspection equipment continues to move forward until the feature object is identified in the depth image before proceeding to the next steps.

[0026] The feature object refers to an object in the alley that is not a wall or road, such as a fork in the road, a person, a vehicle, a sign, a pipe, or a light. If the feature object at the identification point is neither a wall or road nor a fork in the road, a person, a vehicle, a sign, a pipe, or a light, then the feature object is determined to be an abnormal object and the following steps are performed.

[0027] To determine the exact location of the feature object, the following steps can be set: The starting position of the inspection equipment and the speed-time data of the inspection equipment from the starting position to the current time are obtained; when the inspection equipment moves, the speed and time data are recorded in real time and the data is saved, and the speed-time data is retrieved when needed.

[0028] The moving distance of the inspection equipment is obtained based on the speed-time data; wherein a speed curve is obtained based on the speed-time data, and the moving distance of the inspection equipment can be obtained by integrating the speed curve.

[0029] The position of the inspection equipment in the inspection route is obtained based on the moving distance and the starting position. The inspection route is preset using a map of the alleyway to be inspected. The starting position can be preset by marking the inspection route. The moving distance is weighted at the starting position and the moving path of the inspection equipment is obtained based on the preset inspection route. The position of the inspection equipment in the inspection route can be calculated based on the moving path and the inspection route. The position of each feature is obtained based on the position of the inspection device in the inspection route and the distance from each feature to the depth camera. The position of each feature in the inspection route is then calculated by weighting the position of the inspection device in the inspection route and the distance from each feature to the depth camera, and finally by extrapolating based on a preset inspection route.

[0030] When the wide-angle camera 30 takes a picture, the current moment is recorded. Then, following the steps described above, the position of the inspection equipment at this moment is obtained. Based on the current position of the inspection equipment and the distance of the feature object, the position of the feature object in the inspection route is obtained. After obtaining the position of the feature object, it can be stored, transmitted, or used to trigger an alarm, so that maintenance personnel can identify and handle the feature object.

[0031] S3: Using each wide-angle camera 30, acquire a raw image of the distance from each feature to the depth camera 20. The edges of the raw images acquired by adjacent wide-angle cameras 30 overlap.

[0032] In this step, if the inspection equipment is small, it can be considered a base point. The depth camera 20 and multiple wide-angle cameras 30 are all set on this base point. The distance from the depth camera 20 to the feature object can be roughly considered the distance from the base point to the feature object, and the distance from the base point to the feature object can be considered the distance from the multiple wide-angle cameras 30 to the feature object. If the inspection equipment is large, the depth camera 20 and the wide-angle cameras 30 in front of the inspection equipment can be arranged side-by-side, so the distance from the depth camera 20 to the feature object can also be considered the distance from the wide-angle cameras 30 in front of the inspection equipment to the feature object. The distance from the wide-angle cameras 30 in front of the inspection equipment to the feature object is considered the object distance to be photographed. Then, the focal length of the wide-angle cameras 30 is determined based on the object distance so that the wide-angle cameras 30 in front of the inspection equipment focus on the feature object. Simultaneously, the other wide-angle cameras 30 use the same focal length as the wide-angle cameras 30 in front of the inspection equipment to take pictures. This avoids image ghosting caused by different focal lengths during image stitching.

[0033] The inspection equipment is equipped with a control unit 50 for controlling the speed of the inspection equipment. When multiple wide-angle cameras 30 are taking pictures, the control unit 50 on the inspection equipment can be used to control the inspection equipment to slow down or stop, so as to avoid the movement of the inspection equipment causing the image to be blurry. Figure 4 The dashed circle represents multiple wide-angle cameras 30 simultaneously acquiring a ring image. The radius of the dashed circle is the object distance of the multiple wide-angle cameras 30. Two dashed circles represent multiple wide-angle cameras 30 acquiring two sets of images at two different object distances.

[0034] If the inspection equipment is large and requires precise photography, multiple wide-angle cameras 30 can be arranged around an origin point, with the distance from the multiple wide-angle cameras 30 to the origin point being the same. This avoids ghosting caused by positional differences when the wide-angle cameras 30 take pictures at the same object distance.

[0035] If the depth image contains multiple features, each feature corresponds to an object distance. Based on the object distance, the corresponding focal length is obtained. For each focal length, a set of original images is obtained using multiple wide-angle cameras 30. For example... Figure 4 As shown, the depth image contains two features: a human body and a fork in the road. The object distances of the human body and the fork in the road are obtained separately, and the focal lengths corresponding to the human body and the fork in the road are determined. A set of original images is obtained through multiple wide-angle cameras 30 according to the focal length corresponding to the human body, and a set of original images is obtained through multiple wide-angle cameras 30 according to the focal length corresponding to the fork in the road. In this way, a set of original images is obtained for each feature, which can ensure that each feature has sufficient clarity.

[0036] S4: Stitch together multiple original images of the distance from each feature to the depth camera 20 to obtain several panoramic stitched images. The stitching process is as follows: S41: For multiple original images of the distance from each feature to the depth camera 20, obtain matching point pairs between adjacent original images. This is achieved by matching adjacent original images using the SIFT algorithm.

[0037] The SIFT algorithm exhibits strong robustness in low-light environments. Specifically, it extracts keypoints from two adjacent original images, locates these keypoints, determines feature orientation, describes the keypoints, and searches for neighboring image neighborhoods. By comparing the feature vectors of each keypoint pair, it identifies several matching feature point pairs. The mapping relationship between these feature points establishes a correspondence between objects, thus completing the search for matching point pairs. SIFT is a computer vision algorithm used to detect and describe local features in images. It finds extreme points in scale space and extracts their position, scale, and rotation invariants. The description and detection of local image features can help identify objects. SIFT features are based on points of interest in the local appearance of objects and are independent of image size and rotation. It also has high tolerance for changes in lighting, noise, and slight perspective. Based on these characteristics, these features are highly salient and relatively easy to extract. In large feature databases, objects are easily identified with few false positives. SIFT feature descriptions also have a high detection rate for partially occluded objects; sometimes only three or more SIFT object features are sufficient to calculate their position and orientation. With current computer hardware speeds and small feature databases, the recognition speed can approach real-time computation. SIFT features contain a large amount of information, making them suitable for fast and accurate matching in massive databases.

[0038] S42: Remove mismatched points from the matching point pairs using the RANSAC algorithm to obtain valid matching point pairs.

[0039] RANSAC is an inspection algorithm that cleans data acquired from images. It is particularly suitable for roadways where elements are mostly regular geometric shapes, such as cylinders and spheres. Mismatches are identified as high-frequency information other than pre-defined features such as cylindrical pipes, cylindrical corners, circular signs, and circular light fixtures.

[0040] The steps of the RANSAC algorithm are as follows: Assume the proportion of valid matching points in the data is t.

[0041] When the computational model uses K points, the case where at least one of the selected points is a mismatch is... So, the probability P of being able to sample the correct N points and calculate the correct model is...

[0042] Thus, the number of iterations of the RANSAC algorithm can be obtained.

[0043] To determine the number of valid matching points, K represents the number of mismatches, and K represents the number of iterations. Valid matches are called interior points, and mismatches are called exterior points.

[0044] In general, an accuracy of 95% or higher is sufficient to meet the precision requirements. Given that there are fewer interior point values ​​in a roadway scenario, the number of iterations required to achieve the same accuracy should be reduced or increased proportionally.

[0045] S43: Stitch together multiple original images of the distance from each feature to the depth camera 20 based on valid matching points to obtain several panoramic stitched images. The steps include: The global homography matrix H between two adjacent original images is obtained based on the valid matching points:

[0046] Using the DLT algorithm to solve for the global homography matrix H, expanding the above equation yields:

[0047]

[0048] The transformation yields matrix A:

[0049] Its optimization objective is:

[0050] Divide one of the adjacent original images into a grid and obtain the center point of each grid. Calculate the Euclidean distance between the center point of each grid and the valid matching point; The weight W is calculated based on the Euclidean distance. Substitute the weights W into matrix A to obtain the W*A matrix; The W*A matrix is ​​decomposed using the SVD algorithm:

[0051]

[0052] We get x = That is, the smallest singular value corresponds to Given the feature vectors, find the local minimum for each small cell. The local homography matrix of the current mesh is obtained:

[0053] Traverse each grid and map all local homography matrices onto the panoramic canvas to obtain a panoramic stitched image.

[0054] Where A is an arbitrary matrix of size m×n; Let A be the transpose of matrix A; U is an m×m unitary matrix after SVD decomposition. V is an m×n matrix, with all elements except those on the main diagonal being 0; V is an n×n unitary matrix. These are the mapping parameters; x is the horizontal coordinate of the input pixel; y is the vertical coordinate of the input pixel. This represents the horizontal coordinates of the pixel after perspective transformation. This represents the vertical coordinates of the pixel after perspective transformation; The objective is to optimize the Cartesian product of matrix A and matrix h. for The square of the element at position i×i on the diagonal of the matrix; For the V matrix corresponding A vector of position.

[0055] The above scheme employs multiple grids to perform correspondence calculations on information in the image, and uses a set of calculation results to stitch multiple images together, reducing computational load and accelerating processing speed while ensuring accuracy. This scheme is the APAP algorithm, an image stitching algorithm that uses a linear translational transform (DLT) for calculation. The APAP method uses a local transformation model for image stitching. Local models have higher degrees of freedom and are more flexible, better handling local deformations. Compared to global methods, its registration is more accurate, and stitching performance is improved. However, some registration errors and ghosting phenomena still exist in the stitched results. Therefore, local models have better applicability than global models and can register images more accurately.

[0056] S5: Perform feature monitoring on each panoramic stitched image to obtain environmental features within the panoramic stitched image. The steps are as follows: A convolutional neural network model is constructed. The convolutional neural network model is obtained by acquiring an image dataset from the alleyway, using YOLOv3 to train traffic information such as pedestrians and pipelines, and performing model transfer of the alleyway scenario.

[0057] Environmental elements refer to the shape and pixel information of the feature object in the panoramic stitched image. For example, when the feature object is a pipe, the environmental element is the pipe image in the panoramic stitched image.

[0058] Environmental elements in panoramic stitched images are detected using a convolutional neural network model. These environmental elements are categorized according to Table 1, as shown below: Table 1. Environmental elements to be identified in the tunnels

[0059] S6: The panoramic stitched image is compressed using the Seam Carving algorithm to obtain a compressed image. The compressed image is easier to transmit in poor network environments such as alleyways or to be stored with a smaller capacity.

[0060] The Seam Carving algorithm includes the following steps: Calculate image energy in panoramic stitched images The formula is: ; in, This represents the absolute value of the gradient in the x-direction of the panoramic stitched image. Let A and B be the absolute value of the gradient in the y-direction of the panoramic stitched image, where A and B are preset coefficients, A+B=2, and A>B; Calculate the cost map and path map based on image energy; Find and remove the seams with the lowest energy in the panoramic stitched image based on the cost map and path map. Repeat the steps in the Seam Carving algorithm until the preset number of times is reached to obtain the compressed image.

[0061] The Seam Carving algorithm described above is a conventional algorithm in this field. Conventional Seam Carving algorithms calculate image energy. The formula is: ; In this application, when calculating image energy, the conventional image energy... Based on the formula, coefficients A and B are introduced. Common features in roadways include intersections, people, landmarks, pipes, and light fixtures. These features, except for light fixture information, are primarily vertically distributed rather than horizontally distributed. This means that the images of these features have more information in the y-direction and less in the x-direction. Therefore, this application uses coefficient settings to retain more information in the x-direction and reduce information in the y-direction, thus ensuring that the image information of vertically distributed features is not excessively lost during compression. Non-vertically distributed light fixture information is more noticeable in low-light environments, and even reducing information in the y-direction will not lead to excessive loss of light fixture information. Through the seam carving algorithm, a certain number of seams are processed to effectively control a large number of irrelevant low-frequency elements in the image. This controls the image size while retaining image feature information and high-frequency information, efficiently ensuring that the system equipment still has a long working time in weak network environments. Its purpose is to remove information with low information content from the image while retaining key elements to reduce the image size. In real-world scenarios, there are many minor, irrelevant factors, such as walls without any equipment information and edge scenes without any elements. By using seam algorithms, the main elements can be preserved to a large extent, and the image size can be reduced more efficiently.

[0062] In the above scheme, since the inspection equipment is located in the roadway and the GPS signal is poor, it is difficult to directly obtain the location and moving distance of the inspection equipment. Therefore, the inspection equipment is set to inspect according to a fixed route. At the same time, the moving distance of the inspection equipment is obtained by integrating the moving speed and moving time to obtain the approximate position of the inspection equipment in the inspection route. Then, the distance of the feature in the depth image can represent the position of the feature relative to the inspection equipment. Then, by weighting the position of the inspection equipment and the distance of the feature, the position of the feature in the inspection route can be obtained. Then, the position is recorded so that maintenance personnel can find the feature in the inspection route in time for maintenance work.

[0063] To avoid interference from the tunnel environment during information transmission, date and time information can also be obtained, and the distance to the feature object, the date and time information when the original image was acquired, the original image, the panoramic stitched image, the compressed image, environmental elements, and the position of the feature object in the inspection route can be stored. This application also provides an inspection device, including: Vehicle body 10; Depth camera 20, which is located at the front of the vehicle body with the lens facing forward; The wide-angle camera 30 includes at least one first wide-angle camera 30 and a second wide-angle camera 30 located on both sides of the vehicle's direction of movement. The first wide-angle camera 30 faces the front or rear of the vehicle, and the second wide-angle camera 30 faces the side of the vehicle. The image processing unit 40 is used to implement the steps of the above-described image processing method when executing a computer program; Control unit 50 is used to control the movement of the inspection equipment.

[0064] In the above scheme, after the multiple wide-angle cameras 30 acquire the corresponding original images with distance as the object distance, the control unit 50 can slow down or stop the movement of the inspection equipment to avoid ghosting of the original images acquired by the wide-angle cameras 30 due to excessive movement speed.

[0065] The inspection equipment is also equipped with an information storage unit and an information transmission unit for storing and sending information to facilitate information processing by staff. Its information transmission unit provides an RJ45 network interface, a USB interface, and an HDMI display interface, as well as a corresponding management module, enabling the display of streaming data and the transmission of stored image files.

[0066] In summary, this invention uses a depth camera 20 to acquire the location of a feature in real time and then obtains the corresponding focal length. The wide-angle camera 30 adjusts its focal length to obtain an original image with clear feature information. In this way, when multiple wide-angle cameras 30 acquire images corresponding to the same feature, they only need to acquire one image with the same focal length. This ensures the clarity at the location of the feature while avoiding image ghosting caused by different focal lengths in multiple images corresponding to the same feature.

[0067] Throughout this description, numerous specific details, such as examples of components and / or methods, are provided to provide a complete understanding of embodiments of this application. However, those skilled in the art will recognize that embodiments of this application may be practiced without one or more of these specific details or by other devices, systems, components, methods, parts, materials, components, etc.

[0068] It should also be understood that one or more of the elements shown in the figures may be implemented in a more separate or more integrated manner, or may even be removed because they are inoperable in certain circumstances or provided because they may be useful for a particular application.

[0069] Furthermore, unless otherwise expressly stated, any arrows in the accompanying drawings should be considered illustrative only and not limiting. Additionally, unless otherwise stated, the term "or" as used herein is generally intended to mean "and / or". Where a term is anticipated to provide a capability of separation or combination that is unclear, a combination of components or steps will also be considered as indicated.

[0070] The above description of the embodiments shown in this application (including the content set forth in the abstract of the specification) is not intended to be an exhaustive enumeration or to limit this application to the precise forms disclosed herein. Although specific embodiments and examples of this application have been described herein for illustrative purposes only, various equivalent modifications are possible within the spirit and scope of this application, as will be recognized and understood by those skilled in the art. As indicated, these modifications can be made to this application in accordance with the above description of the embodiments described herein, and such modifications will be within the spirit and scope of this application.

[0071] This document has generally described the systems and methods in detail to aid in understanding the present application. Furthermore, various specific details have been provided to offer a general understanding of the embodiments of this application. However, those skilled in the art will recognize that embodiments of this application can be practiced without one or more specific details, or using other means, systems, accessories, methods, components, materials, parts, etc. In other instances, well-known structures, materials, and / or operations have not been specifically shown or described in detail to avoid obscuring various aspects of the embodiments of this application.

[0072] Therefore, although this application has been described herein with reference to specific embodiments thereof, freedom of modification, various changes and substitutions are also within the scope of the above disclosure, and it should be understood that in some cases, certain features of this application may be adopted without departing from the scope and spirit of the invention and without corresponding use of other features. Thus, many modifications can be made to adapt a particular environment or material to the essential scope and spirit of this application. This application is not intended to be limited to the specific terminology used in the following claims and / or the specific embodiments disclosed as the best mode of carrying out this application, but this application will include any and all embodiments and equivalents falling within the scope of the appended claims. Therefore, the scope of this application will be determined only by the appended claims.

Claims

1. An image processing method, characterized in that, Inspection equipment including a depth camera and multiple wide-angle cameras includes the following steps: The depth image in front of the inspection equipment is obtained through the depth camera of the inspection equipment; The depth image is subjected to feature identification to obtain several features and the distance of each feature to the depth camera; Using each of the wide-angle cameras, a raw image of the distance from each feature to the depth camera is acquired, wherein the edges of the raw images acquired by the wide-angle cameras at adjacent locations overlap; Multiple original images of the distance from each feature to the depth camera are stitched together to obtain several panoramic stitched images; Element monitoring is performed on each of the panoramic stitched images to obtain the environmental elements in the panoramic stitched images; The step of performing feature recognition on the depth image to obtain several features and the distance of each feature to the depth camera further includes: obtaining inspection equipment location information, and obtaining the position of the feature based on the inspection equipment location information; The step of obtaining the location information of the inspection equipment and obtaining the location of the feature based on the location information of the inspection equipment includes: Obtain the starting position of the inspection equipment and the speed-time data of the inspection equipment from the starting position to the current time; The moving distance of the inspection equipment is obtained based on the speed-time data; The position of the inspection equipment in the inspection route is obtained based on the moving distance and the starting position; The position of each feature is obtained based on the position of the inspection device in the inspection line and the distance of each feature from the depth camera.

2. The image processing method according to claim 1, characterized in that, In the step of monitoring each of the panoramic stitched images to obtain the environmental elements in the panoramic stitched images, the environmental elements include one or more of the following: intersections, people, vehicles, signs, pipes, and lights.

3. The image processing method according to claim 1, characterized in that, The step of stitching together multiple original images of the distance from each of the aforementioned features to the depth camera to obtain several panoramic stitched images includes: For each of the original images representing the distance from the feature to the depth camera, obtain matching point pairs between adjacent original images; Remove erroneous matches from matching point pairs to obtain valid matches; Multiple original images of the distance from each feature to the depth camera are stitched together based on valid matching points to obtain several panoramic stitched images.

4. The image processing method according to claim 3, characterized in that, The step of obtaining matching point pairs between adjacent original images from multiple original images of the distance from each feature to the depth camera includes: For each feature and the distance to the depth camera, multiple original images are used to match adjacent original images with wide-angle cameras using the SIFT algorithm to obtain matching point pairs between adjacent original images.

5. The image processing method according to claim 3, characterized in that, The step of removing erroneous matches from matching point pairs to obtain valid matches includes: The RANSAC algorithm is used to remove mismatched points from the matching point pairs in order to obtain valid matching points.

6. The image processing method according to claim 5, characterized in that, The step of stitching together multiple original images of the distance from each feature to the depth camera based on valid matching points to obtain several panoramic stitched images includes: The global homography matrix H between two adjacent original images is obtained based on the valid matching points; The global homography matrix H is solved using the DLT algorithm to obtain matrix A; Divide one of the adjacent original images into a grid and obtain the center point of each grid. Calculate the Euclidean distance between each of the grid center points and the valid matching points; Calculate the weight W based on the Euclidean distance; Substitute the weight W into the matrix A to obtain the W*A matrix; The local homography matrix of the current mesh is obtained by decomposing the W*A matrix using the SVD algorithm; Traverse each grid and map all the local homography matrices onto the panoramic canvas to obtain the panoramic stitched image.

7. The image processing method according to claim 1, characterized in that, After stitching together multiple original images of the distance from each of the features to the depth camera to obtain several panoramic stitched images, the method further includes the following steps: The panoramic stitched image is compressed using the Seam Carving algorithm to obtain a compressed image; The Seam Carving algorithm includes the following steps: Calculate the image energy in the panoramic stitched image. The formula is: ； in, This represents the absolute value of the gradient in the x-direction of the panoramic stitched image. Let A and B be the absolute value of the gradient in the y-direction of the panoramic stitched image, where A and B are preset coefficients, A+B=2, and A>B; Calculate the cost map and path map based on image energy; Find and remove the seams with the lowest energy in the panoramic stitched image based on the cost map and path map. Repeat the steps in the Seam Carving algorithm until the preset number of times is reached to obtain the compressed image.

8. An inspection device, characterized in that, include: Vehicle body; A depth camera, located at the front of the vehicle body; The wide-angle camera includes at least one first wide-angle camera and a second wide-angle camera located on both sides of the vehicle's direction of movement. The first wide-angle camera faces the front or rear of the vehicle, and the second wide-angle camera faces the side of the vehicle. An image processing unit is configured to implement the image processing method as described in any one of claims 1-7 to identify environmental elements in front of the inspection equipment; The control unit is used to control the movement of the inspection equipment.

Images