Patrolling method and device, electronic equipment and storage medium

Abstract

The application provides a method and device for inspection, electronic equipment and storage medium. The method is applied to an inspection device on the terminal side. The method comprises: collecting inspection images and inertial data; performing noise reduction processing on the inspection images based on the inertial data; extracting low-level features of the inspection images after noise reduction processing; uploading the low-level features to the cloud to extract high-level features from the low-level features through the cloud, and determining an inspection result based on the high-level features. The hierarchical algorithm of the cloud provides sufficient computing power for multi-task city inspection, has strong scalability, and has low requirements for the computing power of the inspection device, thereby reducing the cost of the inspection device and being suitable for portable devices.

Description

Technical Field

[0001] This application relates to the fields of image processing and urban inspection technology, and in particular to an inspection method, device, electronic device and storage medium. Background Technology

[0002] Urban inspections are becoming increasingly important in order to prevent accidents involving urban pipelines and ensure urban safety.

[0003] Urban patrols encompass a wide range of activities, including road patrols to detect violations, road damage, and flooding, as well as environmental patrols such as monitoring litter and the proper handling of clothes drying. The diverse targets of urban patrols necessitate significant computational power from the patrol equipment.

[0004] Common urban inspection devices often require the use of large vehicles, such as cars or large special vehicles, to carry the electronic pan-tilt cameras of the inspection device and the workstations on the large vehicles to achieve real-time image acquisition and analysis.

[0005] The aforementioned urban inspection devices are costly, and because they rely on large vehicles, they cannot pass through narrow roads for inspection, thus limiting the inspection scenarios.

[0006] Therefore, there is an urgent need to provide a miniaturized urban inspection device. Summary of the Invention

[0007] This application provides an inspection method, device, electronic device, and storage medium, which deploys the complex analysis of the acquired inspection images in the cloud, reducing the computing power requirements of the inspection device and providing a foundation for the miniaturization of the inspection device.

[0008] Firstly, this application provides an inspection method applied to an inspection device, the method comprising:

[0009] Collect inspection images and inertial data;

[0010] Based on the inertial data, the inspection image is subjected to noise reduction processing;

[0011] Extract low-level features from the denoised inspection image;

[0012] The low-level features are uploaded to the cloud so that high-level features can be extracted from the low-level features in the cloud, and the inspection results are determined based on the high-level features.

[0013] Secondly, this application provides another inspection method applied in the cloud, which includes:

[0014] The low-level features uploaded by the inspection device are obtained by the inspection device extracting features from the image after denoising the collected inspection image based on the collected inertial data.

[0015] High-level features are obtained by extracting features from the low-level features;

[0016] Based on the aforementioned high-level features, the inspection results are determined and output.

[0017] Thirdly, this application provides another inspection method for use in an inspection device, the method comprising:

[0018] Collect inspection images and inertial data;

[0019] Based on the inertial data, the inspection image is denoised, and the denoised inspection image is uploaded to the cloud so that the cloud can extract the low-level features of the denoised inspection image, extract high-level features from the low-level features, and determine the inspection result based on the high-level features.

[0020] Fourthly, this application provides an inspection device, including a camera, an inertial sensor, an image enhancement module, a low-level feature extraction module, and an interaction module;

[0021] The image enhancement module is used to perform noise reduction processing on the inspection images captured by the camera based on the inertial data collected by the inertial sensor.

[0022] The low-level feature extraction module is used to extract low-level features from the noise-reduced inspection image.

[0023] The interaction module is used to upload the low-level features to the cloud, so that the cloud can extract high-level features from the low-level features and determine the inspection results based on the high-level features.

[0024] Fifthly, this application provides an electronic device, comprising:

[0025] A processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the methods provided in any aspect of this application.

[0026] Sixthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the inspection method provided in any aspect of this application.

[0027] Seventhly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the inspection method provided in any aspect of this application.

[0028] The inspection method, apparatus, electronic device, and storage medium provided in this application are designed for urban inspection application scenarios with a wide variety of tasks. During inspection, the inspection device is responsible for acquiring inspection images and inertial data, and performing noise reduction processing on the inspection images based on the inertial data to reduce motion noise caused by the jitter of portable inspection devices, thereby improving the signal-to-noise ratio of the inspection images. When the inspection device has sufficient computing resources, it can also extract low-level features from the images and upload these features to the cloud for subsequent processing. This transfers most of the computing power to the cloud, providing a foundation for miniaturization of the inspection device and reducing its cost. Furthermore, the method by which the inspection device interacts with the cloud to obtain low-level features has lower requirements for network stability and bandwidth, further reducing the overall system cost. Attached Figure Description

[0029] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0030] Figure 1 A schematic diagram illustrating an application scenario provided in an embodiment of this application;

[0031] Figure 2 A flowchart illustrating an inspection method provided in an embodiment of this application;

[0032] Figure 3 A flowchart illustrating another inspection method provided in an embodiment of this application;

[0033] Figure 4 A flowchart illustrating another inspection method provided in this application embodiment;

[0034] Figure 5 A schematic diagram of an urban inspection method provided in an embodiment of this application;

[0035] Figure 6 This is a schematic diagram of the structure of an inspection system provided in an embodiment of this application;

[0036] Figure 7 This is a schematic diagram of another inspection system provided in an embodiment of this application;

[0037] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0038] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0039] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0040] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0041] Figure 1 This is a schematic diagram illustrating an application scenario provided in an embodiment of this application, such as... Figure 1 As shown, when conducting city patrols, the patrol device can be worn on the patrol personnel's body or held in their hand, or it can be mounted on small vehicles such as electric bicycles or motorcycles ridden by the patrol personnel. Figure 1 For example, the device is worn on the shoulder of inspection personnel. Inspection personnel carry the device to conduct city inspections, using the device's camera to collect images or videos of the inspected area.

[0042] To reduce the computational burden on inspection devices, most of the computing power required for urban inspections is deployed in the cloud. The inspection device only needs to perform preliminary analysis of the collected inspection images, such as preliminary feature extraction (referred to as low-level feature extraction), to obtain the low-level features of the inspection images. Before feature extraction, the inspection device can also perform preprocessing on the inspection images, such as image enhancement and image noise reduction. The inspection device uploads the extracted low-level features to the cloud, where subsequent tasks are completed, including high-level feature extraction, detection, classification, and segmentation, to fulfill the corresponding urban inspection tasks, such as road damage identification, lane or road surface segmentation, violation identification, license plate recognition, facial recognition, and waste type identification.

[0043] Because portable devices are unstable and prone to shaking, motion shadows may appear in the inspection images. In order to better remove motion shadows, in some embodiments, the inspection device is equipped with an inertial sensor. The inertial data collected by the inertial sensor is used to remove motion shadows from the inspection images, thereby enhancing the images.

[0044] Compared to traditional large-scale inspection devices, this application provides a portable and miniaturized inspection device that transfers most of the computing power to the cloud, alleviating the problem of excessive computing power in inspection devices, providing a foundation for miniaturization of inspection devices, reducing the cost of inspection devices, and expanding inspection scenarios. At the same time, considering the unstable nature of portable devices, inertial data is combined for inspection image noise reduction, which improves the signal-to-noise ratio of the image and thus improves the accuracy of image detection.

[0045] Figure 2 This is a flowchart illustrating an inspection method provided in an embodiment of this application. The inspection method can be executed by an electronic device with corresponding data processing capabilities, such as an inspection device. The inspection device can be a portable device, such as a wearable device or a handheld device.

[0046] In some embodiments, the inspection device can be installed at the front end of small vehicles such as electric vehicles or motorcycles, such as at the handlebars or basket.

[0047] In some embodiments, the inspection device is about the size of a mobile phone.

[0048] like Figure 2 As shown, this inspection method includes the following steps:

[0049] Step S201: Collect inspection images and inertial data.

[0050] During inspections, the inspection device collects inspection images based on cameras and inertial data based on inertial sensors, such as gyroscopes.

[0051] The acquisition frequency (frame rate) of the inspection images can be 25fps, 30fps, 60fps, or other frequencies. The acquisition frequency of inertial data is higher than that of the inspection images, and can be at the level of hundreds of Hz or kHz, such as 100Hz, 1kHz, 8kHz, or other values.

[0052] In some embodiments, the acquisition frequency of inspection images can be between 25 fps and 60 fps. The acquisition frequency of inertial data needs to be at least 1 kHz to meet the needs of urban inspection.

[0053] To ensure the quality of detection results, such as effectively identifying small objects like trash, the camera resolution should be at least 2K, meaning a high-definition camera should be used.

[0054] Step S202: Based on the inertial data, perform noise reduction processing on the inspection image.

[0055] The inspection device's data processing module can acquire inspection images captured by the camera and inertial data collected by the inertial sensor, and perform noise reduction processing on the inspection images based on the inertial data to remove noise caused by the vibration of the inspection device.

[0056] Specifically, the sampling time of the inertial data and the inspection image can be aligned first to obtain the inertial data corresponding to the inspection image. The sampling time is the time when the data processing module receives the inspection image or inertial data. Since the output frequency of the inertial sensor is much higher than the frame rate of the camera, the inertial data with the sampling time closest to the video frame (one frame of inspection image) can be directly selected as the inertial data corresponding to the inspection image. Noise reduction processing is then performed on the inspection image based on the corresponding inertial data.

[0057] In some embodiments, the inspection image can be denoised based on a deep learning model, such as a Generative Adversarial Network (GAN), according to the corresponding inertial data.

[0058] Because deep learning models have high computational complexity and require high computing power from inspection devices, in order to reduce computational complexity, in some other embodiments, the camera imaging process can be regarded as a linear projection. Based on the corresponding inertial data, the imaging process is modeled to obtain an imaging model. Based on the imaging model, the offset of pixels in the inspection image is corrected to obtain a noise-reduced inspection image.

[0059] Step S203: Extract low-level features from the denoised inspection image.

[0060] Both low-level features and subsequent high-level features are features extracted from the inspection images. High-level features are obtained by further extracting features from low-level features. The naming convention is only used to distinguish different image features.

[0061] Low-level features are typically generalized and easy to express, such as image texture, color, and edges; they are usually unrelated to specific business logic. High-level features, on the other hand, are more complex and difficult to express; they are usually related to business logic.

[0062] The aforementioned low-level features can be extracted based on one or more stages of a pre-trained image detection model. This image detection model is a stacked neural network model, constructed by stacking multiple stages.

[0063] The pre-trained image detection model can be split into smaller parts. The first one or more stages of the image detection model can be deployed in an inspection device, such as in the data processing module, while the remaining stages can be deployed in the cloud. Low-level features can then be extracted based on the first one or more stages of the image detection model deployed in the inspection device.

[0064] For example, the image detection model can be a YOLO (You Only Look Once) model, a RestNet (Residual Neural Network) model, an SSDNet (State Space Decomposition Neural Network) model, or other models.

[0065] Based on the type of image detection model, the preset number of stages to be deployed on the inspection device can be determined, such as 1, 2, or other numbers. Based on the number of stages deployed on the inspection device, the image detection model can be split into two parts: one part deployed on the inspection device and the other part deployed in the cloud. The part deployed on the inspection device consists of the preset number of stages preceding the image detection model.

[0066] A pre-established correspondence between image detection model types and preset quantities can be established. Based on the type of image detection model used and the correspondence, the preset quantity of the image detection model deployed in the inspection device stage can be determined.

[0067] Step S204: Upload the low-level features to the cloud so that high-level features can be obtained by extracting features from the low-level features through the cloud, and determine the inspection results based on the high-level features.

[0068] The inspection device can upload low-level features to the cloud via networks such as the Internet of Things and the Internet. The cloud receives these low-level features, performs further feature extraction to obtain high-level features, and determines the inspection results based on the extracted high-level features. For example, it outputs the target detection results of the inspection image to complete the corresponding business, such as identifying illegal vehicles, license plate numbers, garbage, and faces in the inspection image, or detecting whether there is water accumulation or road damage on the road surface in the inspection image.

[0069] The cloud can extract low-level features from the inspection image based on the high-level stages of the image detection model, obtaining high-level features. Then, based on one or more task heads of the image detection model, it obtains the corresponding detection results according to the high-level features. Finally, by combining the detection results output by each task head, it generates the inspection result for that frame of the inspection image. The high-level stages are the stages remaining in the image detection model after removing the previous one or more stages deployed on one side of the inspection device.

[0070] For example, low-level feature interaction can be achieved through the MQTT (Message Queuing Telemetry Transport) channel, that is, the low-level features extracted by the inspection device can be transmitted to the cloud through the MQTT channel.

[0071] By splitting the image detection model and deploying only a small number of stages on the inspection device, the computing power of the inspection device is greatly saved. Taking YOLOv5-s as an example, 90% of the computing power can be saved, which makes it possible to miniaturize the inspection device.

[0072] The inspection method provided in this application is designed for urban inspection application scenarios with a wide variety of tasks. When performing inspections, the inspection device is responsible for collecting inspection images and inertial data, and performing noise reduction processing on the inspection images based on the inertial data to reduce motion noise caused by the easy shaking of portable inspection devices and improve the signal-to-noise ratio of the inspection images. When the inspection device has sufficient computing resources, it can also be responsible for extracting low-level features of the images and uploading the extracted low-level features to the cloud for subsequent processing. This realizes the transfer of most computing power to the cloud, provides a basis for the miniaturization of the inspection device, and reduces the cost of the inspection device. At the same time, the way the inspection device interacts with the cloud to obtain low-level features has low requirements for network stability and bandwidth, which reduces the overall cost of the system.

[0073] Figure 3 This is a flowchart illustrating another inspection method provided in this application embodiment. This embodiment further refines steps S202 and S203 based on the above embodiment. In this embodiment, the low-level feature extraction layer of the image detection model is deployed on the inspection device, while the high-level feature extraction layer of the image detection model and the task header are deployed in the cloud, such as... Figure 3 As shown, the inspection method provided in this embodiment may include the following steps:

[0074] Step S301: Collect inspection images and inertial data.

[0075] Step S302: Based on the inertial data, determine the rotation vector of the camera during the exposure time of acquiring the inspection image.

[0076] The rotation vector R(t) is used to represent the relationship between the camera offset and the offset of pixels in the inspection image.

[0077] The motion trajectory of the camera can be determined based on inertial data, and then the offset of each pixel in the inspection image can be determined based on the motion trajectory of the camera, thus obtaining the rotation vector R(t).

[0078] Based on the sampling time corresponding to the inertial data and the sampling time of the inspection image, the inertial data corresponding to the inspection image can be determined, and based on the inertial data corresponding to the inspection image, the rotation vector R(t) corresponding to the inspection image can be determined.

[0079] Optionally, the inertial data includes the angular velocity collected by the gyroscope. Based on the inertial data, the rotation vector of the camera is determined within the exposure time of acquiring the inspection image, including:

[0080] During the exposure time of acquiring the inspection image, the angular velocity is integrated along the time axis to obtain the rotation vector.

[0081] Angular velocity is used to describe the rate of change of an angle.

[0082] By integrating the angular velocity corresponding to the inspection image along the time axis during the exposure time of the inspection image, the rotation vector R(t) corresponding to the inspection image can be obtained, that is, R(t)=∫Ω(t)dt, where Ω(t) is the angular velocity corresponding to the inspection image.

[0083] Step S303: Based on the rotation vector, perform noise reduction processing on the inspection image.

[0084] The correspondence between the camera movement and the pixel movement in the inspection image can be determined by the rotation vector R(t). Based on this correspondence, the pixels in the inspection image are corrected to obtain the noise-reduced inspection image, thereby filtering out noise caused by camera shake, such as motion shadows, in the inspection image.

[0085] The pixels in the inspection image can be corrected based on the inverse operation of this correspondence.

[0086] Optionally, based on the rotation vector, the inspection image is subjected to noise reduction processing, including:

[0087] Based on the rotation vector and the internal parameters of the camera, the pixels in the inspection image are corrected to obtain a noise-reduced image.

[0088] The internal parameters K of the camera include the camera's focal length f, dx, dy, u0, v0, etc. dx represents the actual length occupied by a pixel in the x-direction, dy represents the actual length occupied by a pixel in the y-direction, and u0 and v0 represent the number of pixels in the x-direction (horizontal coordinate) and y-direction (vertical coordinate) that differ between the pixel coordinates of the image center and the pixel coordinates of the image origin, respectively.

[0089] The camera's internal parameters are used to determine the mapping relationship between the camera and the acquired inspection images in a static state. By combining the camera's internal parameters and this rotation vector, the correlation between the camera's motion and the cloud-based relationships of pixels in the acquired inspection images can be determined during camera movement.

[0090] The process of camera imaging can be regarded as a linear projection. During the movement of the camera, the transformation of its projection can be represented by the following transformation matrix H(t):

[0091] H(t) = KR(t) / K -1

[0092] R(t)=∫Ω(t)dt

[0093] Where K is a constant matrix corresponding to the internal parameters of the camera, which can be directly calibrated.

[0094] Based on the transformation matrix H(t) mentioned above, the pixel offset caused by the camera's movement can be described as follows:

[0095] X ′ =H(t)X

[0096] Among them, X ′ X represents the position of any pixel in the inspection image; X represents the position of the pixel after correction based on the transformation matrix H(t).

[0097] The following formula can be used to correct the pixels in the inspection image to remove motion shadows and obtain a noise-reduced inspection image:

[0098] X=K(∫Ω(t)dt) -1 / K -1 X ′

[0099] By treating the imaging process as a linear projection, and based on the angular velocity collected by the gyroscope and a simple relationship, the motion trail of the inspection image can be removed. The processing algorithm is simple, the computing power requirement of the inspection device is low, and the cost of the inspection device is further reduced.

[0100] Step S304: Input the noise-reduced inspection image into the low-level feature extraction layer of the pre-trained image detection model to obtain the low-level features.

[0101] The paradigm of this image detection model includes multiple stages. The low-level feature extraction layer comprises one or more of the previous stages of the image detection model, while the subsequent high-level feature extraction layer comprises the remaining stages. The pre-trained image detection model is split into two parts: one part is deployed on the inspection device, including the low-level feature layer, and the other part is deployed in the cloud, including the high-level feature extraction layer and the task header.

[0102] The image detection model can be pre-trained based on a training sample set until the training termination condition is met, and the trained image detection model can be obtained through testing.

[0103] After the noise reduction of the inspection image is completed, the noise-reduced inspection image is transmitted to the low-level feature extraction layer deployed on the inspection device. The low-level feature extraction layer extracts features from the noise-reduced inspection image to obtain low-level features.

[0104] Step S305: Upload the low-level features to the cloud.

[0105] In some embodiments, the inspection device further includes a positioning module, which can upload low-level features and location information output by the positioning module to the cloud.

[0106] In some embodiments, the inspection method may further include steps S306 to S308 executed by the cloud.

[0107] Step S306: The low-level features are extracted through a high-level feature extraction layer deployed in the cloud to obtain high-level features.

[0108] In some embodiments, the low-level features can be tensor quantized before uploading to reduce the amount of data in the low-level features. Then, the quantized low-level features are uploaded to the cloud. The cloud then performs dequantization on the quantized low-level features to obtain the low-level features. Based on the high-level feature extraction layer, the low-level features are extracted to obtain the high-level features.

[0109] Step S307: The model output is obtained through the task header deployed in the cloud, based on the high-level features.

[0110] The number of task headers can be one or more, to detect different targets, such as lanes, road water accumulation, road damage, illegally parked vehicles, license plate numbers, faces, clothing, etc.

[0111] The high-level feature extraction layer inputs the extracted high-level features into one or more task heads, which then determine the detection results based on these features. The model output includes the detection results from each task head.

[0112] Step S308: The detection results are obtained and output via the cloud based on the model output.

[0113] Based on the model output, the cloud generates detection results according to the set format and sends the detection results to the business side so that relevant business personnel can take response measures based on the detection results, such as clearing the water on the road surface in the detection results and deducting points from illegally parked vehicles.

[0114] In some embodiments, the cloud can merge the detection results of multiple inspection images and send the merged detection results to the business end.

[0115] In this embodiment, by deploying the low-level feature extraction layer of the image detection model on the inspection device and the high-level feature extraction layer and task header on the cloud, the inspection device only needs to perform low-level feature extraction through the low-level feature extraction layer, saving the inspection device's computing power. Feature extraction through the model is efficient, accurate, and has strong generalization ability and good robustness. By treating the camera imaging process as a linear projection and combining it with the angular velocity output by the gyroscope, a rotation vector is obtained. The offset of the pixels during the camera movement is obtained through the rotation vector. By correcting this offset, an inspection image with motion shadow removed is obtained. The image denoising algorithm is simple and requires less computing resources, further reducing the cost of the inspection device and enabling the miniaturization of the inspection device.

[0116] Figure 4 This is a flowchart illustrating another inspection method provided in this application. This embodiment further refines step S202 based on the above embodiment, adding a step before step S202 to perform time pairing of inertial data and inspection images, and a step to determine the camera's angle change; and adding a step after step S203 to quantize low-level features. For example... Figure 4 As shown, the inspection method provided in this embodiment may include the following steps:

[0117] Step S401: Collect inspection images and inertial data.

[0118] Step S402: Based on the acquisition time, perform time alignment on the inertial data and the inspection image to obtain the inertial data corresponding to each frame of the inspection image.

[0119] Based on the acquisition time or sampling time, the inertial data closest to the time of the inspection image can be selected from the inertial data to become the inertial data corresponding to the inspection image, thus achieving time alignment between the inertial data and the inspection image.

[0120] In some embodiments, inertial data acquired at a time corresponding to the exposure time of the inspection image can be selected as relational data corresponding to the inspection image.

[0121] Step S403: Determine the angle of change of the camera based on the inertial data corresponding to the inspection image.

[0122] For each frame of the inspection image, the angle of camera change is determined based on the inertial data corresponding to that inspection image.

[0123] Taking inertial data as angular velocity as an example, the angle that the camera changes when acquiring the inspection image frame can be determined by integrating the angular velocity along the time axis.

[0124] Step S404: Determine whether the angle of change of the camera is greater than the angle threshold; if yes, proceed to step S405; if no, proceed to step S406.

[0125] The angle threshold is a configurable parameter, typically a small angle, such as less than or equal to 15°.

[0126] When the camera changes angle too much, exceeding the set angle threshold, noise reduction processing is required for the corresponding frame of the inspection image. Conversely, the noise reduction process can be skipped, and low-level features of the inspection image can be extracted directly.

[0127] By setting the angle threshold, the number of noise reduction processes is reduced, further saving the computing power of the inspection device.

[0128] Optionally, the method further includes:

[0129] The angle threshold is determined based on the inspection target.

[0130] Specifically, the angle threshold can be determined based on the type or size of the inspection target. The larger the size of the inspection target, the larger the angle threshold.

[0131] In some embodiments, the angle threshold may include two selectable values, 5° and 10°. When the target being inspected is a small target, such as garbage or license plates, the angle threshold is 5°, while when the target being inspected is a large target, such as vehicles or pedestrians, the angle threshold is 10°.

[0132] By differentiating the angle threshold settings, the accuracy of the angle threshold setting is improved, the number of noise reduction processes is reasonably reduced, the computing power of the inspection device is further saved, the power consumption of the inspection device is reduced, and the working time of the inspection device is extended.

[0133] Step S405: Based on the inertial data, perform noise reduction processing on the corresponding inspection image.

[0134] When the camera changes an angle greater than a set angle threshold during the acquisition of inspection images, noise reduction processing is performed on the frame of inspection image based on the corresponding inertial data, and then step S406 is executed to extract low-level features of the noise-reduced inspection image.

[0135] Step S406: Extract low-level features from the inspection image.

[0136] When the camera changes an angle less than or equal to a set angle threshold while collecting inspection images, the noise reduction process can be skipped, and the low-level features of the inspection image frame can be directly extracted. Specifically, the low-level features of the inspection image frame are extracted based on the low-level feature extraction layer of the pre-trained image detection model deployed on the inspection device.

[0137] Step S407: Perform tensor quantization on the low-level features to upload the quantized low-level features to the cloud.

[0138] In the field of neural networks, the features extracted by the model are called tensors. Tensor quantization is a technique used to perform computations and store tensors with a bit width lower than floating-point precision; it is a data compression technique. For example, it quantizes floating-point values ​​in a tensor into integers. Tensor quantization can yield more compact feature representations, such as reducing tensors by 2 to 4 times, thereby reducing bandwidth requirements.

[0139] A range-based linear quantization algorithm can be used to quantize low-level features, resulting in quantized low-level features. These quantized low-level features are then uploaded to the cloud via an IoT platform, where the cloud performs inverse quantization on the quantized tensor to obtain the low-level features. A high-level feature extraction layer deployed in the cloud then extracts features from these low-level features to obtain high-level features. Based on these high-level features, the model output is obtained through one or more task headers, and the detection results, such as a detection report, are obtained based on the model data.

[0140] In this embodiment, based on the sampling time, the inertial data and the inspection images are time-aligned to obtain the inertial data corresponding to each frame of the inspection images. Based on the inertial data corresponding to the inspection images, the angle of the camera change when the inspection images are acquired is determined. When the angle of change is greater than a set angle threshold, the inspection images are denoised. By setting the angle threshold to filter the inspection images, the number of denoising processes is reduced, and power consumption is reduced. After extracting the low-level features of the inspection images, the low-level features are quantized before being uploaded to the cloud, which reduces the amount of data interacting with the cloud, reduces the requirements for bandwidth and network stability, and reduces system costs.

[0141] This embodiment provides another inspection method, executed by the cloud, such as by one or more servers in the cloud. The method includes:

[0142] The inspection device uploads low-level features, which are obtained by the inspection device extracting features from the image after denoising the collected inspection image based on the collected inertial data; high-level features are obtained by extracting features from the low-level features; and the inspection result is determined and output based on the high-level features.

[0143] In some embodiments, the inspection device can directly upload the noise-reduced inspection image to the cloud, where a complete image detection model deployed in the cloud performs feature extraction and analysis on the noise-reduced inspection image to obtain model output. Based on the model output, the inspection result is obtained.

[0144] This embodiment provides another inspection method, executed by an inspection device, which includes:

[0145] Collect inspection images and inertial data; based on the inertial data, perform noise reduction processing on the inspection images, and upload the noise-reduced inspection images to the cloud so that the cloud can extract low-level features of the noise-reduced inspection images, perform feature extraction on the low-level features to obtain high-level features, and determine the inspection results based on the high-level features.

[0146] Since no inspection device is needed to extract features from the inspection images, the computational requirements for the inspection device are further reduced. However, compared to using an inspection device and cloud-based interactive images, this method... Figure 2 This method requires high network stability and bandwidth.

[0147] In practical applications, the characteristics of the inspection device can be used to determine whether the inspection device should perform low-level feature extraction. For example, if the inspection device has sufficient computing resources, or if the network stability of the area inspected by the inspection device is poor, or if the system bandwidth is low, then one or more stages of the image detection model can be deployed in the inspection device, so that the inspection device is responsible for low-level feature extraction. Figure 2 The inspection is performed using the method shown. If the inspection device has limited computing resources, the network in the inspection area is stable, and the system bandwidth is high, the inspection device can directly upload the noise-reduced inspection image to the cloud, where the cloud will be responsible for tasks such as feature extraction and target detection.

[0148] Figure 5 This is a schematic diagram of an urban inspection method provided in an embodiment of this application, as shown below. Figure 5As shown, inspection personnel ride patrol vehicles along designated routes to conduct city inspections. These vehicles can be small vehicles such as bicycles, electric bikes, or motorcycles to facilitate inspections of narrow alleyways. A single inspection device can handle one or more inspection tasks, including but not limited to road flooding detection, facial recognition, illegal parking detection, garbage sorting, road surface segmentation, and lane marking recognition.

[0149] The inspection device is mounted on the inspection vehicle, such as at the handlebars. During inspection, the device includes a camera, a gyroscope, an image processing module, and a communication module. The camera acquires inspection images at a set frequency, and the gyroscope acquires inertial data at a set frequency. The image processing module is responsible for denoising the inspection images based on the inertial data and extracting low-level features from the denoised images. The communication module is responsible for uploading the low-level features to the cloud server.

[0150] Since the inspection device is responsible for extracting low-level features, such as local point and line features in the image, which are irrelevant to specific tasks or business, the task of extracting high-level features is moved to the cloud, which greatly reduces the workload of the inspection device and provides a basis for the miniaturization of the inspection device.

[0151] Because the inspection device is mounted on a moving inspection vehicle, motion shadows are generated in the acquired inspection images as the vehicle moves. To better remove these motion shadows, this application provides a motion shadow removal method based on inertial data. Specifically, the rotation vector of the camera acquiring the inspection image is obtained by integrating the inertial data angular velocity along the time axis during the exposure time. Based on this rotation vector, noise reduction processing is performed on the inspection image to remove the motion shadows and improve the signal-to-noise ratio of the inspection image.

[0152] The cloud server receives low-level features uploaded by each inspection device. Based on the powerful processing capabilities of the cloud, it performs further feature extraction on the low-level features to obtain high-level features. Based on multiple business-specific headers, it performs classification, detection, and other operations on the high-level features for the detection business of each detection device, and obtains and outputs the detection results, such as the detection results of urban inspection tasks such as road water accumulation recognition, face recognition, and illegal parking vehicle recognition.

[0153] Figure 6 This is a schematic diagram of the structure of an inspection system provided in an embodiment of this application, as shown below. Figure 6 As shown, the inspection system includes multiple portable inspection devices deployed on the edge and a data processing platform deployed in the cloud. Figure 6 Take two portable inspection devices as an example.

[0154] The portable inspection device includes an image acquisition module, an inertial sensing module, an image enhancement module, and a low-level feature extraction module. The image acquisition module includes a camera for acquiring inspection images; the inertial sensing module includes an inertial sensor, such as a gyroscope, for acquiring inertial data; the image enhancement module performs time alignment between the inertial data and the inspection images based on the sampling time, and estimates and eliminates motion blur caused by camera shake in the inspection images based on the corresponding inertial data to improve the signal-to-noise ratio of the inspection images; the low-level feature extraction module extracts low-level features from the inspection images and can also perform quantization processing on these low-level features.

[0155] The data processing platform includes multiple business-specific high-level feature extraction modules, business-specific detection modules, business-specific classification modules, business-specific segmentation modules, and an event center module. Business-specificity refers to the use of different modules for processing different business needs. The high-level feature extraction module receives low-level features and performs further feature extraction to obtain high-level features. The detection, classification, and segmentation modules are all task headers; based on different business needs, one or more task headers can be selected to process and analyze high-level features, obtaining the corresponding model output. The event center module is a business encapsulation module used to obtain detection results based on the model output, generate serialized event information based on multiple detection results, and push the serialized event information to the business end, such as the city operation center or city dashboard.

[0156] For example, the pedestrian recognition service can correspond to a high-level feature extraction module and a detection module to detect pedestrians in the inspection image; the lane line recognition service can correspond to a high-level feature extraction module, a detection module, and a classification module to identify lane lines and their line types in the inspection image; and the road water accumulation service can correspond to a high-level feature extraction module and a detection module to detect water accumulation areas on the road in the inspection image.

[0157] For multi-task urban inspection application scenarios, image analysis under each task can be performed in parallel through the high-level feature extraction module and task header corresponding to each task, so as to obtain the model output corresponding to each task.

[0158] The system provides motion noise removal capabilities through an image enhancement module and provides the computing power expansion and parallel analysis capabilities required for complex analysis scenarios through a cloud-integrated hierarchical algorithm module, enabling the inspection system to handle urban inspections in multiple tasks and scenarios.

[0159] Figure 7This is a schematic diagram of another inspection system provided in this application embodiment. To address the limitation of computing power in portable inspection devices, this embodiment provides a multi-stage cloud-integrated model detection solution, reducing the algorithm's dependence on the computing power of portable inspection devices throughout the entire process, and simultaneously effectively reducing the system's network bandwidth consumption. Figure 7 As shown, the inspection system responsible for feature extraction and analysis includes, in sequence: a low-level feature extractor, tensor quantizer, and MQTT communication pipeline deployed on the edge; and an MQTT communication pipeline, tensor dequantizer, business-specific high-level feature extractor, and task-specific header deployed in the cloud. This header may include classification headers, detection headers, segmentation headers, etc. The MQTT communication pipelines between the edge and cloud communicate through an MQTT server. The MQTT server (broker) is responsible for receiving and forwarding MQTT messages between the edge and cloud MQTT communication pipelines to achieve communication between the cloud and the edge, thereby sending the quantized low-level features to the cloud.

[0160] By splitting the multi-stage neural network model (such as the image detection model mentioned above), the first one or two stages of the neural network model can be calculated on the edge. With the help of the Internet of Things platform, the hierarchical features are sent to the cloud for more complex subsequent calculations to obtain the final detection result.

[0161] After removing motion shadows, or before denoising, the inspection image is first processed by a low-level feature extractor to obtain low-level features. These features are used to describe general features such as points, lines, and grayscale distribution in the image. They are usually features without specific business meanings, and multiple types of business or tasks can share this low-level feature extractor.

[0162] The low-level feature is quantized using a tensor quantizer, such as quantizing a float32 value to an in8 value, thereby reducing the amount of data by 3 / 4.

[0163] The quantized low-level features are pushed to the cloud via the MQTT communication pipeline.

[0164] Based on the tensor dequantizer, the low-level features after quantization are dequantized to obtain the low-level features, thus realizing tensor restoration.

[0165] Based on the high-level feature extractor, high-level stage calculations are performed on low-level features according to the specific business requirements to obtain business-related high-level features. Then, through the corresponding task headers, such as detection headers, classification headers, and segmentation headers, the high-level features are used for final forward calculations to obtain the model output. At this point, the calculation of the relevant parts of the model is completed.

[0166] In some embodiments, detection results can be obtained based on the model output and then sent to the business side.

[0167] In other embodiments, multiple model outputs can be packaged and sent directly to the business side.

[0168] This application provides an inspection device, which includes: a camera, an inertial sensor, an image enhancement module, a low-level feature extraction module, and an interaction module. The image enhancement module is used to perform noise reduction processing on the inspection image captured by the camera based on the inertial data collected by the inertial sensor. The low-level feature extraction module is used to extract low-level features from the noise-reduced inspection image. The interaction module is used to upload the low-level features to the cloud so that high-level features can be extracted from the low-level features through the cloud, and the inspection result can be determined based on the high-level features.

[0169] In some embodiments of the inspection device, the low-level feature extraction module can be omitted, and the noise-reduced inspection image can be directly uploaded to the cloud through the interaction module.

[0170] Optionally, the low-level feature extraction module is specifically used for:

[0171] The denoised inspection image is input into the low-level feature extraction layer of a pre-trained image detection model to obtain the low-level features.

[0172] The high-level feature extraction layer and the task header of the image detection model are deployed in the cloud. The high-level feature extraction layer is used to extract features from the low-level features to obtain high-level features, and input the high-level features into the task header to obtain the model output, so as to obtain the detection result based on the model output.

[0173] Optional image enhancement modules include:

[0174] A rotation vector determination unit is used to determine the rotation vector of the camera during the exposure time of acquiring the inspection image based on the inertial data; a noise reduction processing unit is used to perform noise reduction processing on the inspection image based on the rotation vector.

[0175] Optionally, the inertial data includes angular velocity acquired by a gyroscope, and the rotation vector determination unit is specifically used for:

[0176] During the exposure time of acquiring the inspection image, the angular velocity is integrated along the time axis to obtain the rotation vector.

[0177] Optional, noise reduction processing unit, specifically used for:

[0178] Based on the rotation vector and the internal parameters of the camera, the pixels in the inspection image are corrected to obtain a noise-reduced image.

[0179] Optionally, the device further includes:

[0180] The time alignment module is used to perform time alignment between the inertial data and the inspection image based on the acquisition time, so as to obtain the inertial data corresponding to each frame of the inspection image.

[0181] The angle change determination module is used to determine the angle of change of the camera based on the inertial data corresponding to the inspection image;

[0182] The image enhancement module is specifically used for:

[0183] When the angle of change corresponding to the inertial data is greater than the angle threshold, noise reduction processing is performed on the corresponding inspection image based on the inertial data.

[0184] Optionally, the device further includes:

[0185] An angle threshold determination module is used to determine the angle threshold based on the inspection target.

[0186] Optionally, the device further includes:

[0187] The quantization processing module is used to perform tensor quantization on the low-level features so as to upload the quantized low-level features to the cloud.

[0188] The inspection device provided in this application can be used to execute the technical solution of the inspection method performed by the inspection device in any embodiment of this application. The implementation principle and technical effect are similar, and will not be described again in this embodiment.

[0189] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, such as... Figure 8 As shown, the electronic device 800 provided in this embodiment includes:

[0190] At least one processor 810; and a memory 820 communicatively connected to the at least one processor; wherein the memory 820 stores computer-executable instructions; the at least one processor 810 executes the computer-executable instructions stored in the memory to cause the electronic device 800 to perform the inspection method provided in any of the foregoing embodiments.

[0191] Optionally, the memory 820 can be either standalone or integrated with the processor 810.

[0192] The electronic device can be either the server or the client mentioned above.

[0193] The implementation principle and technical effects of the electronic device 800 provided in this embodiment can be found in the foregoing embodiments, and will not be repeated here.

[0194] This application also provides a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are executed by a processor, the methods provided in any of the foregoing embodiments can be implemented.

[0195] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the method provided in any of the foregoing embodiments.

[0196] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed.

[0197] The integrated modules implemented as software functional modules described above can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this application.

[0198] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor. The memory may include high-speed memory, and may also include non-volatile memory, such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk, or optical disc, etc.

[0199] The aforementioned storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory, electrically erasable programmable read-only memory, erasable programmable read-only memory, programmable read-only memory, read-only memory, magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0200] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. The processor and storage medium can reside within an application-specific integrated circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic device or inspection device.

[0201] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0202] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0203] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods provided in the various embodiments of this application.

[0204] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0205] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. An inspection method, characterized in that, The method is applied to an inspection device, and the method includes: Collect inspection images and inertial data; Based on the acquisition time, the inertial data and the inspection images are time-aligned to obtain the inertial data corresponding to each frame of the inspection images. Based on the inertial data corresponding to the inspection images, the angle of camera change is determined; When the angle of change corresponding to the inertial data is greater than the angle threshold, the inspection image is denoised based on the inertial data. Extract low-level features from the denoised inspection image; The low-level features are uploaded to the cloud so that high-level features can be extracted from the low-level features in the cloud, and the inspection results are determined based on the high-level features. The noise reduction process for the inspection image based on the inertial data includes: The camera imaging process is regarded as a linear projection. Based on the inertial data, the imaging process is modeled to obtain an imaging model. Based on the imaging model, the offset of the pixels in the inspection image is corrected to obtain the noise-reduced inspection image.

2. The method according to claim 1, characterized in that, Low-level features are extracted from the denoised inspection image, including: The denoised inspection image is input into the low-level feature extraction layer of a pre-trained image detection model to obtain the low-level features. The high-level feature extraction layer and the task header of the image detection model are deployed in the cloud. The high-level feature extraction layer is used to extract features from the low-level features to obtain high-level features, and input the high-level features into the task header to obtain the model output, so as to obtain the detection result based on the model output.

3. The method according to claim 1, characterized in that, Based on the inertial data, noise reduction processing is performed on the inspection image, including: Based on the inertial data, the rotation vector of the camera is determined within the exposure time of acquiring the inspection image; The inspection image is denoised based on the rotation vector.

4. The method according to claim 3, characterized in that, The inertial data includes angular velocities acquired by a gyroscope. Based on the inertial data, the rotation vector of the camera is determined within the exposure time of acquiring the inspection image, including: During the exposure time of acquiring the inspection image, the angular velocity is integrated along the time axis to obtain the rotation vector.

5. The method according to claim 3, characterized in that, Based on the rotation vector, the inspection image is subjected to noise reduction processing, including: Based on the rotation vector and the internal parameters of the camera, the pixels in the inspection image are corrected to obtain a noise-reduced image.

6. The method according to claim 1, characterized in that, The method further includes: The angle threshold is determined based on the inspection target.

7. The method according to any one of claims 1-4, characterized in that, The method further includes: The low-level features are subjected to tensor quantization to upload the quantized low-level features to the cloud.

8. An inspection method, characterized in that, The method is applied in the cloud and includes: The low-level features uploaded by the inspection device are obtained by the inspection device extracting features from the image after denoising the collected inspection image based on the collected inertial data. High-level features are obtained by extracting features from the low-level features; The high-level features are input into one or more task heads, which then determine and output inspection results based on the high-level features. The number of task heads can be one or more to detect different targets.

9. An inspection method, characterized in that, The method is applied to an inspection device, and the method includes: Collect inspection images and inertial data; Based on the acquisition time, the inertial data and the inspection images are time-aligned to obtain the inertial data corresponding to each frame of the inspection images. Based on the inertial data corresponding to the inspection images, the angle of camera change is determined; When the angle of change corresponding to the inertial data is greater than the angle threshold, the inspection image is denoised based on the inertial data, and the denoised inspection image is uploaded to the cloud so that the cloud can extract the low-level features of the denoised inspection image, extract high-level features from the low-level features, and determine the inspection result based on the high-level features. The noise reduction process for the inspection image based on the inertial data includes: The camera imaging process is regarded as a linear projection. Based on the inertial data, the imaging process is modeled to obtain an imaging model. Based on the imaging model, the offset of the pixels in the inspection image is corrected to obtain the noise-reduced inspection image.

10. An inspection device, characterized in that, It includes a camera, an inertial sensor, an image enhancement module, a low-level feature extraction module, and an interaction module; The image enhancement module is used to perform noise reduction processing on the inspection images captured by the camera based on the inertial data collected by the inertial sensor. The low-level feature extraction module is used to extract low-level features from the noise-reduced inspection image. The interaction module is used to upload the low-level features to the cloud, so that the cloud can extract high-level features from the low-level features and determine the inspection results based on the high-level features. The image enhancement module is specifically used to: treat the camera imaging process as a linear projection, model the imaging process based on the inertial data to obtain an imaging model, and correct the offset of the pixels in the inspection image based on the imaging model to obtain a noise-reduced inspection image. The image enhancement module is also used to: perform time alignment of the inertial data and the inspection image based on the acquisition time, so as to obtain the inertial data corresponding to each frame of the inspection image; Based on the inertial data corresponding to the inspection images, the angle of change of the camera is determined; When the angle of change corresponding to the inertial data is greater than the angle threshold, the inspection image is denoised based on the inertial data.

11. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the method as described in any one of claims 1-9.

Images