Image detection method, device, computer equipment and storage medium

By dividing and merging regions of interest in an image on the terminal device and using an edge server for object detection, the problem of low real-time performance in mobile vision applications is solved, and efficient object detection results are achieved.

CN114723731BActive Publication Date: 2024-11-22HUNAN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210437675.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-25
Publication Date
2024-11-22
Estimated Expiration
2042-04-25

AI Technical Summary

Technical Problem

Existing mobile vision applications suffer from low real-time performance, especially during driving. Object detection solutions often use low-resolution image processing, resulting in low precision and accuracy, or rely on hardware to improve computing power but still cannot effectively handle large-scale detection tasks.

Method used

By extracting and segmenting the region of interest from image data on the terminal device, merging overlapping regions, and using edge servers for target detection, the combined work of local and edge servers improves computational efficiency and reduces latency.

Benefits of technology

It improves the real-time performance and accuracy of target detection in mobile vision scenarios, reduces latency, and enhances the detection efficiency of terminal devices.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114723731B_ABST
    Figure CN114723731B_ABST
Patent Text Reader

Abstract

The present application relates to an image detection method, device, computer equipment and storage medium. The method includes: acquiring image data, extracting a region of interest from the image data, dividing the region of interest into a first region of interest with an area greater than or equal to a preset area threshold and a second region of interest with an area less than the preset area threshold according to the area of ​​the region of interest, merging overlapping areas in the second region of interest to obtain a third region of interest, performing target detection on the first region of interest, obtaining local detection results, and constructing a target detection task corresponding to the third region of interest, sending the target detection task carrying the third region of interest to an edge server, receiving the detection results fed back by the edge server, merging the local detection results and the detection results fed back by the edge server, and obtaining the final detection results. The use of this method can significantly improve the real-time performance of target detection in mobile visual scenes.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image detection method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] With the continuous development of technology and mobile devices, mobile vision applications have also emerged. Mobile devices such as mobile phones generate a large number of images and videos, which further promotes the technical improvement of mobile vision applications and has higher and higher requirements. It has excellent effects in various aspects such as target recognition, environmental detection, emergency detection, industrial robots, etc. However, during driving, mobile vision has high real-time and low latency requirements, so the effect of mobile vision applications is not ideal.

[0003] At present, the cameras of mobile devices usually acquire high-resolution images, and the target detection schemes for mobile vision often use low-resolution images for processing in order to reduce latency, which is likely to lose effective image information, resulting in low precision and low accuracy. Or a lot of work has been done to improve hardware performance, such as using GPU (Graphic Processing Unit) and FPGA (Field Programmable Gate Array) to improve the computing power of the device, thereby reducing latency, but instability still occurs in mobile vision applications, and it is unable to cope with large-scale detection tasks.

[0004] This shows that the current mobile vision target detection solution has the problem of low real-time performance. Summary of the invention

[0005] Based on this, it is necessary to provide an image detection method, apparatus, computer device, computer-readable storage medium and computer program product that can reduce delay in response to the above technical problems.

[0006] In a first aspect, the present application provides an image detection method, which is applied to a terminal whose computing power is lower than a preset computing power requirement. The method comprises:

[0007] Acquire image data, and extract a region of interest from the image data;

[0008] According to the area of ​​the region of interest, the region of interest is divided into a first region of interest and a second region of interest, the first region of interest being a region of interest having an area greater than or equal to a preset area threshold, and the second region of interest being a region of interest having an area less than the preset area threshold;

[0009] Merging overlapping regions in the second region of interest to obtain a third region of interest;

[0010] Performing target detection on the first region of interest, obtaining a local detection result, and constructing a target detection task corresponding to the third region of interest, and sending the target detection task carrying the third region of interest to the edge server, so that the edge server performs target detection on the received third region of interest;

[0011] Receive detection results fed back by the edge server;

[0012] The local detection results and the detection results fed back by the edge server are combined to obtain the final detection result.

[0013] In one embodiment, extracting the region of interest from the image data includes:

[0014] Extract feature data of image data and generate feature map;

[0015] Perform binary classification processing on the feature map to obtain a binary classification result;

[0016] According to the binary classification results, the background area of ​​the image data is filtered and the region of interest of the image data is extracted.

[0017] In one embodiment, before sending the target detection task carrying the third region of interest to the edge server, the method further includes:

[0018] Determining, according to the area of ​​the third region of interest, a scheduling priority of the target detection task corresponding to the third region of interest;

[0019] Get the scheduling priority of the edge server;

[0020] Sending the target detection task with the third region of interest to the edge server includes:

[0021] Based on the scheduling priority of the target detection task and the scheduling priority of the edge server, the target detection task is sent to the edge server in parallel.

[0022] In one embodiment, determining the scheduling priority of the target detection task corresponding to the third region of interest according to the area of ​​the third region of interest includes:

[0023] sorting the third regions of interest according to their areas to obtain an arrangement order of the third regions of interest;

[0024] According to the arrangement order of the third regions of interest, the scheduling priority of the object detection task corresponding to the third regions of interest is determined.

[0025] In one embodiment, before performing target detection on the first region of interest and obtaining a local detection result, the method further includes:

[0026] Downsampling the first region of interest to obtain a downsampled first region of interest;

[0027] Perform target detection on the first region of interest, and obtain local detection results including:

[0028] The downsampled first region of interest is input into the preset Faster R-CNN (Convolutional Neural Network) DNN (Dynamic Neural Neork) model for target detection, the target detection result output by the preset Faster R-CNN-based DNN model is obtained, and the local detection result is obtained.

[0029] In a second aspect, the present application also provides an image detection device. The device comprises:

[0030] A data extraction module, used to obtain image data and extract a region of interest from the image data;

[0031] A data division module, used for dividing the region of interest into a first region of interest and a second region of interest according to the area of ​​the region of interest, wherein the first region of interest is a region of interest having an area greater than or equal to a preset area threshold, and the second region of interest is a region of interest having an area less than the preset area threshold;

[0032] A data merging module, used for merging the overlapping areas in the second region of interest to obtain a third region of interest;

[0033] An image detection module is used to perform target detection on the first region of interest, obtain a local detection result, and construct a target detection task corresponding to the third region of interest, and send the target detection task carrying the third region of interest to the edge server, so that the edge server performs target detection on the received third region of interest;

[0034] A data receiving module is used to receive the detection results fed back by the edge server;

[0035] The detection result merging module is used to merge the local detection results and the detection results fed back by the edge server to obtain the final detection results.

[0036] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0037] Acquire image data, and extract a region of interest from the image data;

[0038] According to the area of ​​the region of interest, the region of interest is divided into a first region of interest and a second region of interest, the first region of interest being a region of interest having an area greater than or equal to a preset area threshold, and the second region of interest being a region of interest having an area less than the preset area threshold;

[0039] Merging overlapping regions in the second region of interest to obtain a third region of interest;

[0040] Performing target detection on the first region of interest, obtaining a local detection result, and constructing a target detection task corresponding to the third region of interest, and sending the target detection task carrying the third region of interest to the edge server, so that the edge server performs target detection on the received third region of interest;

[0041] Receive detection results fed back by the edge server;

[0042] The local detection results and the detection results fed back by the edge server are combined to obtain the final detection result.

[0043] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0044] Acquire image data, and extract a region of interest from the image data;

[0045] According to the area of ​​the region of interest, the region of interest is divided into a first region of interest and a second region of interest, the first region of interest being a region of interest having an area greater than or equal to a preset area threshold, and the second region of interest being a region of interest having an area less than the preset area threshold;

[0046] Merging overlapping regions in the second region of interest to obtain a third region of interest;

[0047] Performing target detection on the first region of interest, obtaining a local detection result, and constructing a target detection task corresponding to the third region of interest, and sending the target detection task carrying the third region of interest to the edge server, so that the edge server performs target detection on the received third region of interest;

[0048] Receive detection results fed back by the edge server;

[0049] The local detection results and the detection results fed back by the edge server are combined to obtain the final detection result.

[0050] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0051] Acquire image data, and extract a region of interest from the image data;

[0052] According to the area of ​​the region of interest, the region of interest is divided into a first region of interest and a second region of interest, the first region of interest being a region of interest having an area greater than or equal to a preset area threshold, and the second region of interest being a region of interest having an area less than the preset area threshold;

[0053] Merging overlapping regions in the second region of interest to obtain a third region of interest;

[0054] Performing target detection on the first region of interest, obtaining a local detection result, and constructing a target detection task corresponding to the third region of interest, and sending the target detection task carrying the third region of interest to the edge server, so that the edge server performs target detection on the received third region of interest;

[0055] Receive detection results fed back by the edge server;

[0056] The local detection results and the detection results fed back by the edge server are combined to obtain the final detection result.

[0057] The above-mentioned image detection method, apparatus, computer equipment, storage medium and computer program product extract the region of interest of the image data, and divide the region of interest into a first region of interest which can be called a region of interest including large objects and a second region of interest which can be called a region of interest including small objects according to the area of ​​the region of interest. Then, the overlapping areas in the second region of interest are merged to obtain a third region of interest, so that the second region of interest can contain complete object information as much as possible, which can improve the accuracy of target detection. Then, the target detection is directly performed locally on the first region of interest, so that the local detection results can be obtained quickly. At the same time, a target detection task corresponding to the third region of interest is constructed, and the target detection task carrying the third region of interest is sent to the edge server, so as to utilize the high computing power and efficiency of the edge server to perform rapid target detection on the received third region of interest and reduce delay. Therefore, the adoption of this solution can significantly improve the real-time performance of target detection in mobile visual scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 A diagram showing an application environment of an image detection method in an embodiment;

[0059] Figure 2 1 is a schematic diagram of a flow chart of an image detection method in an embodiment;

[0060] Figure 3 is a schematic flow chart of an image detection method in another embodiment;

[0061] Figure 4 is a structural block diagram of an image detection device in one embodiment;

[0062] Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0064] The image detection method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 is a terminal whose computing power is lower than the preset computing power requirement, that is, a terminal without high computing power, and the terminal 102 communicates with the edge server cluster 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the edge server cluster 104, or it can be placed on the cloud or other network servers. Specifically, the terminal 102 may trigger an image detection program in a mobile autonomous visual scene, collect large-scale image data during the movement of the terminal, process the image data, extract the region of interest of the image data, and then divide the region of interest into a first region of interest and a second region of interest according to the area of ​​the region of interest, wherein the first region of interest is a region of interest having an area greater than or equal to a preset area threshold, and the second region of interest is a region of interest having an area less than the preset area threshold, and then, merge the overlapping areas in the second region of interest to obtain a third region of interest, and then directly perform target detection on the first region of interest locally to obtain a local detection result, and construct a target detection task corresponding to the third region of interest, and send the target detection task carrying the third region of interest to the edge server, so that the edge server performs target detection on the received third region of interest, receives the detection result fed back by the edge server, and merges the local detection result and the detection result fed back by the edge server to obtain the final detection result, so as to complete the target detection of the image data. The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices, vehicle-mounted devices, and portable wearable devices, and the Internet of Things devices may be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, and the like. The portable wearable device may be a smart watch, a smart bracelet, a head-mounted device, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0065] In one embodiment, if Figure 2 As shown, an image detection method is provided, which is applied to Figure 1 The terminal in is used as an example to illustrate, including the following steps:

[0066] Step 100: Acquire image data and extract a region of interest from the image data.

[0067] The region of interest refers to ROI (region of interest). In this embodiment, the image data takes a high-resolution image collected in a mobile autonomous visual scene as an example, and generally speaking, the image collected in a mobile autonomous visual scene is a large-scale high-resolution image with a large image size. After acquiring the image data, it is necessary to identify the area that may contain objects in each frame of the image, and then extract the region of interest to perform target detection and accurately identify the category of the object. It is understandable that there are multiple regions of interest extracted. In specific implementation, in order to improve the detection efficiency, the image data can be compressed first to reduce its resolution, and then the region of interest in the image data can be extracted.

[0068] Step 200, dividing the region of interest into a first region of interest and a second region of interest according to the area of ​​the region of interest, the first region of interest being a region of interest having an area greater than or equal to a preset area threshold, and the second region of interest being a region of interest having an area less than the preset area threshold.

[0069] After extracting the region of interest in the image data, the area of ​​the region of interest can be calculated first, and then the region of interest is traversed, and the area of ​​the region of interest is compared with the preset area threshold in turn, and the region of interest with an area greater than or equal to the preset area threshold is determined as the first region of interest, that is, the region of interest containing large objects, and the region of interest with an area less than the preset area threshold is determined as the second region of interest containing small objects. In another embodiment, the regions of interest can also be arranged in descending order according to the area size, and then the sorted regions of interest are compared with the preset area threshold one by one to divide them. It is worth mentioning that, in general, the device needs to perform global target detection on the image data, and the amount of engineering is very large. In this example, the region of interest is divided, so that the target detection model in the edge server only needs to focus on the detection of the third small object of interest in the cropped small block, and the area of ​​the divided region of interest is always less than 20% of the full-size image. The terminal also only needs to focus on the detection of the first region of interest, which also avoids wasting time on the detection of targets in other regions of interest, thereby improving the detection efficiency. In addition, only 20% of the data size is transmitted between the edge level and the edge server, and the detection delay becomes low.

[0070] Step 300: merge overlapping areas in the second region of interest to obtain a third region of interest.

[0071] Since there may be a large number of overlapping areas between the divided regions of interest containing small objects, in order to improve the detection accuracy of small objects, the overlapping areas of the second regions of interest may be first identified, and then the overlapping areas may be merged to obtain the third regions of interest, so that each third region of interest includes complete object information as much as possible. For example, multiple regions of interest containing small objects may contain partial regions of flowers and grass, two small objects. At this time, the partially overlapping areas of flowers and grass may be merged, so that a single second region of interest contains complete flower information, and a single region of interest contains complete grass information. Furthermore, the original regions of interest before the merger may be deleted to avoid redundancy of regions of interest and waste of computing resources.

[0072] Step 400, perform target detection on the first region of interest, obtain local detection results, and construct a target detection task corresponding to the third region of interest, and send the target detection task carrying the third region of interest to the edge server, so that the edge server performs target detection on the received third region of interest.

[0073] Due to the limited computing power of the terminal, it is impossible to cope with the target detection task of large-scale images. Therefore, in this embodiment, the terminal and the edge server can be used to coordinate target detection to improve the efficiency of target detection and reduce delay. Specifically, after completing the merging of the overlapping areas of the second area of ​​interest containing small objects, the DNN model is directly used locally to perform target detection on the first area of ​​interest containing large objects to obtain the detection result. The detection result may include target recognition results such as flowers and trees, and corresponding target detection evaluation indicators such as precision, recall rate, etc. At the same time, the corresponding target detection task is constructed for the third area of ​​interest. It can be that according to the corresponding task scheduling strategy, the target detection task carrying the third area of ​​interest is sent to the edge server, so that the edge server performs target detection on the received third area of ​​interest. In this way, the high computing power and efficiency of the edge server can be used to quickly and accurately detect targets and reduce delays.

[0074] In one embodiment, before performing target detection on the first region of interest and obtaining a local detection result, the method further includes: downsampling the first region of interest to obtain a downsampled first region of interest;

[0075] In order to obtain good target detection performance, the edge server can use a complex target detection model with more parameters and more computing resources, such as a feature pyramid network, to perform target detection on the received second region of interest. In this embodiment, the edge server uses FPN (Feature Pyramid Network) as the backbone of the target detection model for target detection, which can make full use of FPN's bottom-up characteristics, as well as top-down paths and lateral connections, and can combine low-dimensional and high-dimensional spatial features to perform accurate target detection.

[0076] Step 500: receiving the detection result fed back by the edge server.

[0077] Following the above embodiment, after the edge server completes the target detection on the third region of interest, it will feed back the detection result to the terminal, and the terminal receives the detection result fed back by the edge server, thus completing the target detection on the image data.

[0078] Step 600: Combine the local detection result and the detection result fed back by the edge server to obtain the final detection result.

[0079] In a specific implementation, after obtaining the local detection result and the detection result fed back by the edge server, since the detection results at this time are for the first region of interest and the third region of interest, respectively, and are relatively scattered, the local detection result and the detection result fed back by the edge server can be merged to obtain the final detection result, that is, the local detection result and the detection result fed back by the edge server are integrated together to obtain the detection result corresponding to the entire image data. In this embodiment, by merging the local detection result and the detection result fed back by the edge server, the observability of the detection data of the image data can be improved, and it is convenient for the staff to view the detection results.

[0080] In the above-mentioned image detection method, the region of interest of the image data is extracted, and according to the area of ​​the region of interest, the region of interest is divided into a first region of interest, which can be called a region of interest including large objects, and a second region of interest, which can be called a region of interest including small objects. Then, the overlapping areas in the second region of interest are merged to obtain a third region of interest, which can make the second region of interest contain complete object information as much as possible, and can improve the accuracy of target detection. Then, the target detection is directly performed on the first region of interest locally, and the local detection results can be quickly obtained. At the same time, a target detection task corresponding to the third region of interest is constructed, and the target detection task carrying the third region of interest is sent to the edge server, so as to utilize the high computing power and efficiency of the edge server to quickly detect the received third region of interest and reduce delay. Therefore, the adoption of this solution can significantly improve the real-time performance of target detection in mobile visual scenes.

[0081] like Figure 3 As shown, in one embodiment, step 100 includes: step 120, acquiring image data, extracting feature data of the image data, generating a feature map, performing binary classification processing on the feature map to obtain a binary classification result, filtering the background area of ​​the image data according to the binary classification result, and extracting the area of ​​interest of the image data.

[0082] After acquiring the image data, since the image data may be a high-resolution image, the amount of data to be processed is large. Therefore, a pre-trained DNN based on Faster R-CNN can be used to extract the region of interest. Specifically, the backbone network can be used to extract the feature data of the image data and generate a feature map. The feature map can then be input into the RPN (Region Proposal Network) to perform cutting and filtering operations on the feature map to complete the binary classification of the target, and then correct the classification result to obtain the final binary classification result. Then, according to the binary classification result, the background area in the image data is filtered to extract the region of interest. In this embodiment, the region of interest can be quickly and accurately extracted through binary classification processing.

[0083] Perform target detection on the first region of interest, and obtain local detection results including:

[0084] The downsampled first region of interest is input into the preset DNN model based on Faster R-CNN for target detection, the target detection result output by the preset DNN model based on Faster R-CNN is obtained, and the local detection result is obtained.

[0085] In this embodiment, target detection on the first region of interest can be: Similarly, the first region of interest is first downsampled to obtain the downsampled first region of interest, and then the downsampled first region of interest is input into the lightweight DNN based on Faster R-CNN for target detection. The model first classifies the candidate boxes in the first region of interest, and then fine-tunes the coordinates of the classified candidate boxes. Next, the classifier in the lightweight DNN will output the fine-tuning regression information of the coordinates and the classification information of the detection frame through two fully connected layers and two fully connected layers of the same size but without shared weights, respectively, to obtain the detection result. It can be understood that other target detection models can also be used for target detection, which is not limited here. In this embodiment, by downsampling the first region of interest, the resolution of the first region of interest can be reduced, and the effect of target detection by the local terminal can be improved.

[0086] like Figure 3As shown, in one embodiment, step 200 includes: step 220, obtaining the area of ​​the region of interest, comparing the area of ​​the region of interest with preset area thresholds, determining the region of interest whose area is greater than or equal to the preset area threshold as the first region of interest, and determining the region of interest whose area is less than the preset area threshold as the second region of interest.

[0087] In one embodiment, before sending the target detection task carrying the third region of interest to the edge server, the method further includes:

[0088] According to the area of ​​the third region of interest, the scheduling priority of the target detection task corresponding to the third region of interest is determined, and the scheduling priority of the edge server is obtained.

[0089] Sending the target detection task carrying the third region of interest to the edge server includes: sending the target detection task to the edge server in parallel based on the scheduling priority of the target detection task and the scheduling priority of the edge server.

[0090] In order to reasonably schedule the target detection task, the third region of interest can be divided into multiple intervals according to the size of the third region of interest, and then the priority of the third region of interest in each interval is set in units of intervals, and the scheduling priority of the target detection task with the third region of interest is determined accordingly. For example, according to the area of ​​the third region of interest, the third region of interest is divided into 4 intervals, and 4 priorities of the third region of interest are obtained accordingly. Once the priority of the third region of interest is determined, the scheduling priority of the target detection task with the third region of interest is also determined. It is also possible to directly set the corresponding priority according to the size of the area of ​​the third region of interest, which is not limited here.

[0091] In specific implementation, the number of edge servers may be multiple. To schedule the target detection tasks between the edge servers, the scheduling priority of the edge servers may also be set accordingly. In this embodiment, the scheduling priority of the edge server is determined according to the computing power of the edge server. The higher the computing power, the higher the scheduling priority, wherein the computing power is based on the computing power of the edge server to process each unit of image data. Then, based on the scheduling priority of the target detection task and the scheduling priority of the edge server, the target detection task is sent to the edge server in parallel. In this embodiment, an offline method may be used to evaluate the computing power of the edge server per unit of image data. Specifically, the computing power of the edge server may be evaluated by: inputting an evaluated data set D on the edge server, the data set containing x regions of interest less than an area threshold, represented by RoI. For RoI di∈D, 1≤i≤x, assuming that the time when the edge server starts processing is tsi and the time when the detection ends is tei, it can be obtained that the detection delay of di on the edge server is tli=tsi-tei. Therefore, the unit computing power of the edge server to process di is tli / area(di), wherein area(di) represents the area of ​​RoI di. In addition, by averaging the evaluation results of the computing power of the edge server for all RoIs in the data set D, the computing power of the edge server to process each unit region of interest can be estimated. It is assumed that the computational cost is linearly proportional to the area of ​​the region of interest. According to the computing power per unit area of ​​the edge server obtained by offline calculation, it can be obtained that the computational cost of the server to calculate the roi with a total area of ​​A is CP=A×U. Since in this embodiment, it is assumed that the computing power of the edge servers is the same, the computing cost of the edge servers is determined by the area of ​​the region of interest. In this embodiment, by setting the scheduling priority of the target detection task and the scheduling priority of the edge server, the target detection task can be reasonably offloaded to the corresponding edge server to improve the task detection efficiency.

[0092] In one embodiment, determining the scheduling priority of the target detection task corresponding to the third region of interest based on the area of ​​the third region of interest includes: sorting the third region of interest according to the area of ​​the third region of interest to obtain an arrangement order of the third region of interest, and determining the scheduling priority of the target detection task corresponding to the third region of interest based on the arrangement order of the third region of interest.

[0093] In this embodiment, the third regions of interest may be arranged in descending order according to their area to obtain an arrangement order of the third regions of interest, and the scheduling priority of the target detection task corresponding to the third region of interest may be determined in sequence according to the arrangement order, that is, the larger the area, the higher the scheduling priority, and the order of the regions of interest represents their scheduling priority. It is understandable that the ordering method may also be a positive order arrangement or a random order, etc. In this embodiment, the third regions of interest are sorted according to their area, so that the scheduling priority of the target detection task corresponding to the third region of interest can be determined conveniently and reasonably.

[0094] In one of the embodiments, sending the target detection tasks to the edge servers in parallel based on the scheduling priority of the target detection tasks and the scheduling priority of the edge servers includes: sending the target detection tasks to the edge servers in parallel based on the earliest deadline first principle based on the scheduling priority of the target detection tasks and the scheduling priority of the edge servers.

[0095] In order to ensure the detection accuracy of the region of interest and minimize the total delay of the detection. In this embodiment, a dynamic area-aware parallel scheduling (hereinafter referred to as DAPS) algorithm is designed to offload the detection task to the edge server. The DAPS algorithm schedules the detection task according to the earliest deadline priority principle. In this embodiment, similarly, the scheduling priority is set according to the size of the third region of interest. That is, the larger the area of ​​the RoI area, the higher its scheduling priority. The DAPS algorithm schedules the RoI in order from high to low priority. Specifically, based on the third region of interest, a RoI set S is constructed, and the RoI set S passed to the edge server is arranged in descending order according to the area of ​​the area. Therefore, the order of the RoI in the set S also represents the descending order of its priority. For a set of edge servers E = {e1, e2, ..., em}, the DAPS algorithm estimates their computing power U at time t and maintains it using a list L, list L = {}, and records the earliest startup time of the edge server in list L. The value of Lti represents the time required from time t to when the i-th edge server can start a new offload task. The DAPS algorithm schedules the RoI according to the priority of the detection task, reads the computing power of the edge server, and then schedules each target detection task according to the earliest deadline first principle. Specifically, the idea of ​​the DAPS algorithm is: DAPS takes the third region of interest set S = {s1, s2, ..., sn} containing small objects, the edge server E = {e1, e2, ..., em}, and the edge server unit computing power U as input. Then the algorithm will output a scheduling decision to unload the RoI to the edge server. First, initialize an empty list L, then for each edge server ei∈E, 1≤i≤m, add the current computing power U of the edge server ei and the end time of executing the target detection task to the list L, which is also the earliest time that the server ei can execute the new RoI. Since the RoIs in the set S are sorted in descending order of priority, sj in S are accessed in sequence. During the scheduling process, the earliest deadline of the target detection task of the current RoI is always queried. According to the computing power and earliest start time of the edge server, the edge server ei that can execute the current RoI earliest is found, and the target detection task of the RoI is scheduled to the edge server ei. When the edge server unloads a new task, the earliest start time of the next task will change, so after the RoI is scheduled, the value in the list L will be automatically updated to update the earliest start time of the edge server. In another embodiment, assuming that the computing power of the edge servers is the same, the scheduling priority of the edge servers is the same. At this time, the computing power of the edge servers is not considered, and the tasks are directly scheduled according to the earliest deadline of the RoI target detection task and the earliest start time of the edge server, according to the earliest deadline priority principle.In this embodiment, the target detection tasks are scheduled according to the earliest deadline priority principle, so that the target detection tasks can be sent to the edge servers in parallel and can be reasonably sent to the edge servers in the shortest possible time.

[0096] In order to make a clearer description of the image detection method provided by the present application, the method is applied to a mobile device such as a vehicle as an example, and is described in conjunction with a specific embodiment, which is as follows:

[0097] In a mobile autonomous visual scene, the mobile device triggers the image detection program, collects large-scale image data during the terminal movement, and uses the pre-trained lightweight DNN based on Faster R-CNN to process the image data. First, the backbone network is used to extract the feature data of the image data and generate a feature map. Then, the feature map can be input into the RPN (Region Proposal Network) to perform cutting and filtering operations on the feature map to complete the binary classification of the target, and then correct the classification result to obtain the final binary classification result. Then, according to the binary classification result, the background area in the image data is filtered to extract the region of interest. The area of ​​the region of interest is calculated, and then the region of interest is traversed, and the area of ​​the region of interest is compared with the preset area threshold in turn. The region of interest with an area greater than or equal to the preset area threshold is determined as the first region of interest, that is, the region of interest containing large objects, and the region of interest with an area less than the preset area threshold is determined as the second region of interest containing small objects.

[0098] Since there may be a large number of overlapping areas between the divided regions of interest containing small objects, in order to improve the detection accuracy of small objects, the overlapping areas of the second regions of interest may be identified first, and then the overlapping areas may be merged to obtain the third regions of interest, so that each third region of interest includes complete object information as much as possible. After completing the merging of the overlapping areas of all the second regions of interest containing small objects, the terminal and the edge server are used to perform collaborative target detection: on the local side, the first region of interest may be downsampled to reduce its resolution, and then the downsampled first region of interest may be input into the lightweight DNN based on Faster R-CNN for target detection to obtain the detection result. At the same time, the corresponding target detection task is constructed for the third region of interest, and the target detection task carrying the third region of interest may be sent to the edge server according to the corresponding task scheduling strategy. The edge server uses FPN as the backbone of the target detection model to perform target detection on the received third region of interest, and feeds back the detection result to the terminal, receives the detection result fed back by the edge server, and finally, merges the local detection result and the detection result fed back by the edge server to obtain the detection result corresponding to the entire image data.

[0099] In this embodiment, the task scheduling strategy can be obtained according to the DAPS algorithm. Specifically, the scheduling process of the target detection task can be: based on the third region of interest, construct a RoI set S, and arrange the RoI set S passed to the edge server in descending order according to the area of ​​the region. The order of the RoI in the set S also represents the descending order of its priority. For a group of edge servers E = {e1, e2..., em}, estimate their computing power U at time t, and take the same computing power U of the edge servers as an example for explanation. Initialize an empty list L, and then, for each edge server ei∈E, 1≤i≤m, add the current computing power U of the edge server ei and the end time of executing the target detection task to the list L, which is also the earliest time that the server ei can execute the new RoI. Since the RoIs in the set S are sorted in descending order of priority, sj in S are accessed in sequence. During the scheduling process, the earliest deadline of the target detection task of the current RoI is always queried. According to the computing power and earliest start time of the edge server, the edge server ei that can execute the current RoI earliest is found, and the RoI is scheduled to the edge server ei, and the target detection task of the RoI is scheduled to the corresponding edge server ei. When the edge server unloads a new task, the earliest start time of the next task will change, so after scheduling the RoI, the value in the list L will be automatically updated to update the earliest start time of the edge server.

[0100] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0101] Based on the same inventive concept, the embodiment of the present application also provides an image detection device for implementing the above-mentioned image detection method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in one or more image detection device embodiments provided below can refer to the limitations on the image detection method above, and will not be repeated here.

[0102] In one embodiment, if Figure 4As shown, an image detection device is provided, which is applied to a terminal whose computing power is lower than a preset computing power requirement, and includes: a data extraction module 510, a data division module 520, a data merging module 530, an image detection module 540, a data receiving module 550 and a detection result merging module 560, wherein:

[0103] The data extraction module 510 is used to acquire image data and extract the region of interest of the image data.

[0104] The data division module 520 is used to divide the region of interest into a first region of interest and a second region of interest according to the area of ​​the region of interest, wherein the first region of interest is a region of interest having an area greater than or equal to a preset area threshold, and the second region of interest is a region of interest having an area less than the preset area threshold.

[0105] The data merging module 530 is used to merge the overlapping areas in the second region of interest to obtain a third region of interest.

[0106] The image detection module 540 is used to perform target detection on the first region of interest, obtain local detection results, and construct a target detection task corresponding to the third region of interest, and send the target detection task carrying the third region of interest to the edge server so that the edge server performs target detection on the received third region of interest.

[0107] The data receiving module 550 is used to receive the detection result fed back by the edge server;

[0108] The detection result merging module 560 is used to merge the local detection result and the detection result fed back by the edge server to obtain the final detection result.

[0109] The above-mentioned image detection device extracts the region of interest of the image data, and divides the region of interest into a first region of interest which can be called a region of interest including large objects and a second region of interest which can be called a region of interest including small objects according to the area of ​​the region of interest. Then, the overlapping areas in the second region of interest are merged to obtain a third region of interest, which can make the second region of interest contain complete object information as much as possible, and can improve the accuracy of target detection. Then, the target detection is directly performed on the first region of interest locally, and the local detection results can be obtained quickly. At the same time, a target detection task corresponding to the third region of interest is constructed, and the target detection task carrying the third region of interest is sent to the edge server, so as to utilize the high computing power and efficiency of the edge server to perform rapid target detection on the received third region of interest and reduce delay. Therefore, the use of this device can significantly improve the real-time performance of target detection in mobile visual scenes.

[0110] In one embodiment, the data extraction module 510 is also used to extract feature data of the image data, generate a feature map, downsample the feature map to obtain the feature map, perform binary classification processing on the feature map to obtain a binary classification result, filter the background area of ​​the image data based on the binary classification result, and extract the area of ​​interest of the image data.

[0111] In one of the embodiments, the data partitioning module 520 is also used to obtain the area of ​​the region of interest, compare the area of ​​the region of interest with the preset area threshold, determine the region of interest whose area is greater than or equal to the preset area threshold as the first region of interest, and determine the region of interest whose area is less than the preset area threshold as the second region of interest.

[0112] In one of the embodiments, the image detection module 540 is also used to determine the scheduling priority of the target detection task corresponding to the third area of ​​interest based on the area of ​​the third area of ​​interest, obtain the scheduling priority of the edge server, and send the target detection task to the edge server in parallel based on the scheduling priority of the target detection task and the scheduling priority of the edge server.

[0113] In one embodiment, the image detection module 540 is further configured to send the object detection tasks to the edge servers in parallel according to the earliest deadline first principle based on the scheduling priority of the object detection tasks and the scheduling priority of the edge servers.

[0114] In one embodiment, the image detection module 540 is also used to downsample the first region of interest to obtain the downsampled first region of interest, input the downsampled first region of interest into a preset Faster R-CNN DNN model for target detection, obtain the target detection result output by the preset Faster R-CNN-based DNN model, and obtain a local detection result.

[0115] Each module in the above-mentioned image detection device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0116] In one embodiment, a computer device is provided. The computer device may be a terminal whose computing power cannot support large-scale image detection. The internal structure diagram of the computer device may be as follows: Figure 5As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, an image detection method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0117] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0118] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the above-mentioned image detection method when executing the computer program.

[0119] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned image detection method are implemented.

[0120] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above-mentioned image detection method when executed by a processor.

[0121] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0122] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0123] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0124] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the present application. It should be noted that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image detection method, characterized in that: The method comprises: Acquire image data, and extract a region of interest from the image data; According to the area of ​​the region of interest, the region of interest is divided into a first region of interest and a second region of interest, wherein the first region of interest is a region of interest having an area greater than or equal to a preset area threshold, and the second region of interest is a region of interest having an area less than the preset area threshold; Merging overlapping areas in the second regions of interest to obtain a third region of interest; Performing target detection on the first region of interest to obtain a local detection result, and constructing a target detection task corresponding to a third region of interest, and sending the target detection task carrying the third region of interest to an edge server, so that the edge server performs target detection on the received third region of interest; Receiving the detection result fed back by the edge server; The local detection result and the detection result fed back by the edge server are combined to obtain a final detection result.

2. The image detection method according to claim 1, characterized in that: The extracting the region of interest of the image data comprises: Extracting feature data of the image data to generate a feature map; Performing binary classification processing on the feature map to obtain a binary classification result; According to the binary classification result, the background area of ​​the image data is filtered to extract the region of interest of the image data.

3. The image detection method according to claim 1, characterized in that: Before sending the target detection task carrying the third region of interest to the edge server, the method further includes: Determining, according to the area of ​​the third region of interest, a scheduling priority of the target detection task corresponding to the third region of interest; Obtaining the scheduling priority of the edge server; The sending the target detection task carrying the third region of interest to the edge server includes: Based on the scheduling priority of the target detection task and the scheduling priority of the edge server, the target detection task carrying the third region of interest is sent to the edge server in parallel.

4. The image detection method according to claim 3, characterized in that: Determining the scheduling priority of the target detection task corresponding to the third region of interest according to the area of ​​the third region of interest includes: sorting the third regions of interest according to their areas to obtain an arrangement order of the third regions of interest; According to the arrangement order of the third regions of interest, the scheduling priority of the target detection task corresponding to the third regions of interest is determined.

5. The image detection method according to claim 4, characterized in that: The sending the target detection task to the edge server in parallel based on the scheduling priority of the target detection task and the scheduling priority of the edge server comprises: Based on the scheduling priority of the target detection task and the scheduling priority of the edge server, the target detection task is sent to the edge server in parallel according to the earliest deadline first principle.

6. The image detection method according to any one of claims 1 to 5, characterized in that: Before performing target detection on the first region of interest to obtain a local detection result, the method further includes: Downsampling the first region of interest to obtain a downsampled first region of interest; The performing target detection on the first region of interest to obtain a local detection result includes: The downsampled first region of interest is input into a preset DNN model based on Faster R-CNN for target detection, and the target detection result output by the preset DNN model based on Faster R-CNN is obtained to obtain the local detection result.

7. An image detection device, characterized in that: The image detection device comprises: A data extraction module, used to acquire image data and extract a region of interest from the image data; a data division module, configured to divide the region of interest into a first region of interest and a second region of interest according to the area of ​​the region of interest, wherein the first region of interest is a region of interest having an area greater than or equal to a preset area threshold, and the second region of interest is a region of interest having an area less than the preset area threshold; A data merging module, configured to merge overlapping regions in the second regions of interest to obtain a third region of interest; an image detection module, configured to perform target detection on the first region of interest, obtain a local detection result, and construct a target detection task corresponding to a third region of interest, and send the target detection task carrying the third region of interest to an edge server, so that the edge server performs target detection on the received third region of interest; A data receiving module, used to receive the detection result fed back by the edge server; The detection result merging module is used to merge the local detection result and the detection result fed back by the edge server to obtain a final detection result.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.