Image processing method and device, equipment and storage medium

Abstract

The invention provides an image processing method and device, equipment and a storage medium, and relates to the technical field of image processing, in particular to the technical fields of artificial intelligence, computer vision, big data, deep learning and the like. According to the specific implementation scheme, under the condition that a computer vision model judges that a target object in a target image has potential production safety hazards, object features extracted by the computer vision model for the target object are extracted; performing matching operation on the object features and reference features of false detection objects in a white list to obtain a matching result; and under the condition that the object features are matched with the reference features, correcting a detection result of the computer vision model for the target object as that the target object does not have production potential safety hazards.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to technical fields such as artificial intelligence, computer vision, big data, and deep learning. Background Art

[0002] With the continuous development of artificial intelligence, computer vision model (CV) reasoning and recognition technology has been widely used in various fields. The current CV model is basically data-driven and can analyze and process image data to obtain the desired results. Summary of the invention

[0003] The present disclosure provides an image processing method, apparatus, device and storage medium.

[0004] According to one aspect of the present disclosure, there is provided an image processing method, comprising:

[0005] When the computer vision model determines that a target object in the target image has a production safety hazard, extracting object features extracted by the computer vision model for the target object;

[0006] Performing a matching operation on the object features and the reference features of the misdetected objects in the whitelist to obtain a matching result;

[0007] In the case where the object feature matches the reference feature, the detection result of the computer vision model for the target object is corrected to indicate that there is no production safety hazard for the target object.

[0008] According to another aspect of the present disclosure, there is provided an image processing apparatus, comprising:

[0009] An extraction module, for extracting object features extracted by the computer vision model for the target object when the computer vision model determines that the target object in the target image has a production safety hazard;

[0010] A first matching module, used for matching the object features with reference features of misdetected objects in the whitelist to obtain a matching result;

[0011] A correction module is used to correct the detection result of the computer vision model for the target object to indicate that there is no production safety hazard for the target object when the object feature matches the reference feature.

[0012] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0013] at least one processor; and

[0014] a memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0016] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0017] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0018] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0020] Figure 1 is a flowchart of an image processing method according to an embodiment of the present disclosure;

[0021] Figure 2 is a schematic diagram of a target image according to an embodiment of the present disclosure;

[0022] Figure 3 is an architecture diagram of an image processing method according to an embodiment of the present disclosure;

[0023] Figure 4 is another flowchart of an image processing method according to an embodiment of the present disclosure;

[0024] Figure 5 is a schematic diagram of a visualization interface according to an embodiment of the present disclosure;

[0025] Figure 6 is another flowchart of an image processing method according to an embodiment of the present disclosure;

[0026] Figure 7 is a schematic diagram of a scene according to an image processing method in an embodiment of the present disclosure;

[0027] Figure 8 is a schematic diagram of the architecture of an image processing method according to an embodiment of the present disclosure;

[0028] Fig. 9 is a structural schematic diagram of an image processing device according to an embodiment of the present disclosure;

[0029] Fig.10 It is a block diagram of an electronic device used to implement the image processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0031] The terms "first", "second", etc. in this disclosure are used to distinguish similar objects, and are not necessarily used to describe a specific order or precedence. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, including a series of steps or units. The method, system, product or device is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] In actual production environments, computer vision models can be used to detect production environments to ensure production safety. For example, to ensure the safety of workers in production environments, workers’ clothing can be detected, and dangerous work areas can be detected.

[0033] However, production workshops are generally large and have a variety of fixed and movable equipment. Some of the content that needs to be detected may be blocked. Even the outdoor production environment may result in poor quality of the collected images. In short, the computer vision model may misidentify due to light or occlusion. For example, the worker originally wore a helmet, but due to occlusion and poor visibility, the computer vision model identified him as not wearing a helmet.

[0034] In related technologies, in order to reduce the misrecognition rate of computer vision models, the misrecognition data can be recycled and the computer vision model can be retrained using the misrecognition data to reduce the generation of misrecognition data by fine-tuning the model parameters.

[0035] However, the training cycle of computer vision models is too long, wasting a lot of manpower. Retraining also requires additional computing resources. This is not a good solution for the R&D team. Moreover, retraining may not achieve the expected generalization effect, the reasoning effect may not be improved, and the input-output ratio needs to be improved.

[0036] In view of this, an image processing method is proposed in the embodiment of the present disclosure. In the embodiment of the present disclosure, in view of the fact that when the computer vision model is inferring, due to scene problems or boundary conditions, the computer vision model will infer and recognize wrong results. Users can manually add these wrong recognition results to a collection on the application platform to form a whitelist. The use of a whitelist in model post-processing can filter out data that is misidentified by the computer vision model, thereby improving the accuracy of the alarm.

[0037] like Figure 1 FIG. 1 is a flow chart of an image processing method provided by an embodiment of the present disclosure, and includes the following contents:

[0038] S101, when the computer vision model determines that a target object in a target image has a production safety hazard, extracting object features extracted by the computer vision model for the target object.

[0039] The target image may be any image in an actual production environment. Figure 2 As shown, the target object can be a human body in each identification tag frame. In addition, the target object can also be equipment in a production environment, such as a metering pump, wind pressure, etc. The embodiment of the present disclosure does not limit the target object.

[0040] Among them, for human subjects, production safety hazards may be not wearing a safety helmet, performing dangerous operations, etc.

[0041] S102, performing a matching operation on the object features and the reference features of the misdetected objects in the whitelist to obtain a matching result.

[0042] S103, when the object features match the reference features, the detection result of the computer vision model for the target object is corrected to indicate that the target object does not have any production safety hazard.

[0043] In the event that the object features do not match the reference features, it can be confirmed that the production safety hazard identified by the computer vision model is correct, allowing subsequent processes to be executed.

[0044] In the disclosed embodiment, a whitelist is used to assist the computer vision model, and based on the form of matching, it is possible to quickly identify the presence of misidentification and correct it. Since in an actual production environment, the computer vision model is usually input in the form of a video stream, based on the whitelist matching method, it is possible to quickly filter out the results of misidentification and reduce unnecessary alarms. The whitelist can help improve the final alarm efficiency of the computer vision model. The entire process does not require data recovery or retraining of the model, and can be quickly applied to the production environment to improve the accuracy of the alarm. In short, the processing results of the computer vision model can be repaired at a relatively low cost to reduce the false alarm rate.

[0045] Especially in the gradual iteration process of the project, the whitelist post-processing method can ensure that the computer vision model can better serve the entire project. This method is relatively efficient, easy to implement, and has controllable results. It can save the resources required for training the model and will not have a negative impact on the computer vision model and other modules.

[0046] For example, the architecture diagram applicable in the embodiments of the present disclosure is as follows: Figure 3 As shown, it includes a computer vision model, a model package, and a whitelist. The image to be processed is input into the computer vision model to obtain the data structure of the image to be processed. The data structure includes but is not limited to the size of the label box, the coordinates of the upper left corner of the label box, the object features of each target object identified, and the detection confidence and other information. The model package processes the data structure of the target image to obtain the coordinate information, label, object features, etc. of the label box identified for the target object. The model package can be used to visualize the content output by the computer vision model and display specific types of production safety hazards.

[0047] When the computer vision model determines that the image to be processed has production safety hazards, the image to be processed is determined as the target image, and the target object with production safety hazards in the target image is determined. The target object with production safety hazards is subsequently matched with the whitelist, and subsequent processing is performed according to the matching results. Therefore, the alarm accuracy can be improved by rechecking the whitelist.

[0048] Among them, the model package can be one or more. In the case of one model package, the model package can include multiple labels to correspond to multiple production safety hazards. Exemplarily, in the case of detecting potential fire hazard behaviors, the label can include open flames, smoke, and human bodies. In the case of multiple model packages, each production safety hazard can correspond to a model package, or multiple production hazards can correspond to one model package. Exemplarily, in the case where the computer vision model misidentifies a certain production safety hazard 1 more often, the production safety hazard can be corresponded to a model package. In the case where there are fewer misidentifications of production safety hazards 2 and production safety hazards 3, the production safety hazards 2 and production safety hazards 3 can be corresponded to a model package, and the two production safety hazards can be post-processed by one model package to save computing resources. Of course, it can be understood that the embodiments of the present disclosure are not limited to this.

[0049] In some embodiments, Figure 2 As shown, the same target image may include multiple objects, and each object is detected by a computer vision model to determine whether there is a safety hazard. Therefore, in the embodiment of the present disclosure, determining the target object can be implemented as follows: when the computer vision model determines that multiple candidate objects have production safety hazards, obtaining the detection confidence generated by the computer vision model for the multiple candidate objects; based on the detection confidence of the multiple candidate objects, screening out the candidate objects whose offset between the detection confidence and the confidence threshold is less than the offset threshold as the target object.

[0050] During implementation, the exemplary confidence threshold is 60%, the offset threshold is set to 30%, the detection confidence of candidate object 1 is 65%, the detection confidence of candidate object 2 is 78%, and the detection confidence of candidate object 3 is 98%. In this case, the offset of candidate object 1 is 5%, and the offset of candidate object 2 is 18%. Candidate object 1 and candidate object 2 are target objects, respectively, and need to be matched through a subsequent whitelist.

[0051] In the disclosed embodiment, based on the offset between the detection confidence and the confidence threshold, it is possible to determine which candidate objects are target objects, and which need to be subsequently rechecked and filtered based on the whitelist. Thus, by screening out the target objects that need post-processing, the number of queries and matching whitelists is reduced, and based on this approach, computing resources can be saved to a certain extent, while improving recheck efficiency.

[0052] In some embodiments, for a candidate object among the multiple candidate objects whose detection confidence and the offset of the confidence threshold are greater than or equal to the offset threshold, production safety hazard warning information is generated for the candidate object.

[0053] Continuing with the previous example, the offset of candidate object 3 is 38%, which is greater than the offset threshold. In this case, there is a high probability that candidate object 3 has unsafe factors. Therefore, there is no need to pass the whitelist review. A safety hazard alarm message can be directly generated to notify relevant personnel to check and deal with the safety hazard in a timely manner.

[0054] In the disclosed embodiment, since the offset between the detection confidence and the confidence threshold is greater than or equal to the offset threshold for an object, it can be determined that there is a high probability that the object has a security risk. Therefore, there is no need to go through a whitelist check and an alarm information can be directly generated, which can reduce the computing resources required for whitelist matching to a certain extent and improve alarm efficiency.

[0055] In some embodiments, in an actual production environment, the target object may be a mobile target and needs to move in the environment. In the actual production process, in order to improve the accuracy of the alarm, it is necessary to track the situation of the mobile target to detect safety hazards for the mobile target, and generally determine whether the mobile target has safety hazards through multiple frames of images. Therefore, in the case where the target object is a mobile target, the target image is an image in an image group, and the image group is a safety detection performed by the computer vision model on the mobile target, and the detection result of the safety detection is that the motion trajectory and / or motion state of the mobile target meets the safety hazard alarm requirements.

[0056] Among them, when the motion trajectory indicates that it has crossed the boundary or deviated from the original compliant motion route, it is determined that it meets the safety hazard warning requirements; when the motion state represents the state of the target object, there may be safety hazards, such as a person falling down, not wearing a safety helmet, etc.

[0057] In the disclosed embodiment, since the target object is in the process of moving, it is necessary to organize the target object's moving process into an image group and perform security detection on the target objects in the group to ensure the accuracy of the moving target detection, avoid the problem of misidentification due to one-sided detection, and improve the overall efficiency of security detection.

[0058] In some embodiments, for a mobile target, determining its image group can be implemented as follows: tracking and analyzing the target object based on the image to be processed acquired by at least one first image sensor to obtain state characteristics of the target object in multiple frames of first images; matching the state characteristics of the target object in the multiple frames of first images with safety hazard warning requirements; and constructing the image group based on the multiple frames of first images when the state characteristics of the target object in the multiple frames of first images meet the safety hazard warning requirements.

[0059] During implementation, the ID (Identity document) of the same target object in the images to be processed of a preset number of frames can be set to the same ID. The state features of the target object in each frame of the image are extracted. Exemplarily, the state features of the target object in three consecutive frames in time sequence can be extracted through a computer vision model. These three frames are respectively used as the first image to construct an image group. The state features of the target object in the three first images are matched with the safety hazard alarm requirements respectively; when the state features in the first images of three consecutive frames meet the safety hazard alarm requirements, it is determined that the target object has a production safety hazard, so as to facilitate the subsequent whitelist matching operation.

[0060] In the disclosed embodiment, the target object is tracked so that the state characteristics of the target object form an overall image group, so that the entire image group can be rechecked using a whitelist later to improve the alarm accuracy for moving targets.

[0061] In some embodiments, when at least one image in the image group matches a reference feature, an alarm message generated for the image group is filtered out.

[0062] During implementation, continuing with the above example, after constructing the three frames of first images as an image group corresponding to the target object, a matching operation is performed with the reference features of the falsely detected objects in the whitelist to obtain a matching result; if any one of the three frames matches the reference features, the alarm message generated for the image group is filtered out; if no reference features are matched in any of the three frames, an alarm message for the image group is generated.

[0063] In the disclosed embodiment, the whitelist can improve the alarm accuracy of mobile targets to reduce unnecessary alarms.

[0064] In some embodiments, in order to further improve the post-processing efficiency and determine the target image that needs to be re-inspected, it can be specifically implemented as follows: based on the safety detection result of the target object in the second image captured by the multiple second image sensors, when it is found that there is a production safety hazard and the credibility of the detection result is less than the credibility threshold, the target image is screened out from the second images captured by the multiple second image sensors.

[0065] The safety area may be set based on actual conditions, and the embodiments of the present disclosure do not limit this.

[0066] During implementation, due to the angular position of the second image sensor, or light and weather reasons, the image quality is low, resulting in the production safety detection results of some target objects being that there are production safety hazards. In this case, the second image with potential safety hazards can be screened out as the target image for subsequent matching operations with the whitelist. During implementation, whether to filter the target image based on credibility can be determined according to the angular position and / or light and weather conditions of the second image sensor at the time. For example, if the images captured by the second image sensor at a certain angular position are mostly obstructed, the operation of filtering the target image based on credibility can be initiated. For another example, the second image sensor in an outdoor environment collects poor image quality in thunderstorms and foggy weather, and the operation of filtering the target object based on credibility can be initiated. Correspondingly, it can be understood that the reference features of the misdetected objects in this case are also present in the whitelist.

[0067] In the disclosed embodiment, the credibility of the second image is considered, that is, the quality of the second image is taken into consideration. When the credibility is low, the target image is screened out and a subsequent whitelist matching operation is performed, which can reduce unnecessary resource loss to a certain extent.

[0068] In some embodiments, determining the credibility of the security detection result of the target object may be specifically implemented as follows:

[0069] Based on multiple second image sensors, images of the safety area are collected; based on the importance of the multiple second image sensors, corresponding weights are assigned to the second images collected by each second image sensor; based on the second images collected by each second image sensor, corresponding weights are assigned, and the detection results of production safety detection performed by the computer vision model on the target object in the second images collected by the multiple second image sensors are weighted summed to obtain the credibility of the safety detection results of the target object.

[0070] For example, for a scene, a main second image sensor and two second image sensors may be included. The weight of the second image collected by the main second image sensor may be set to 60%, and the weight of the second image collected by the two auxiliary second image sensors may be set to 20%. If the detection result of the second image collected by the main second image sensor is 80, there is a production safety hazard, and the detection results of the second images collected by the two auxiliary second image sensors are 75 and 72 respectively, then the credibility of the safety detection result of the target object is (60%*80+20%*75+20%*72)=77.4, which is then compared with the credibility threshold to perform subsequent processing.

[0071] In the embodiment of the present disclosure, considering that the importance of each second image sensor is different, corresponding weights can be assigned to them to improve the accuracy of security detection of the image. The specific weight setting can be based on actual conditions, and the embodiment of the present disclosure does not limit this.

[0072] In another embodiment, determining the credibility of the safety detection result of the target object can also be implemented as follows: performing safety detection on the target object in the second image captured by the multiple second image sensors based on the computer vision model; and determining the proportion of the second image sensors that are judged to have production safety hazards as the credibility.

[0073] Exemplarily, when there are five second image sensors, if the second images captured by three second image sensors all determine that the target object has safety hazards, the ratio is 60%; when the second images captured by two second image sensors are determined to have production safety hazards, the credibility is determined to be 40%, and then the credibility threshold is further compared. When it is lower than the credibility threshold, the target image is screened out from the second images captured by the multiple second image sensors.

[0074] In some embodiments, filtering out the target image from the second images captured by the multiple second image sensors can be implemented as follows: performing quality evaluation on the image portion of the target object in the second images captured by the multiple second image sensors to obtain image quality; and filtering out the second images whose image quality meets preset requirements as the target image.

[0075] The preset requirement may be lower than a preset quality score.

[0076] During implementation, the image quality assessment model can be used to evaluate the quality of the image and output an image quality score. When the quality score is lower than a preset quality score, it is used as a target image.

[0077] Any method that can determine the image quality is applicable to the embodiments of the present disclosure, and the embodiments of the present disclosure are not limited to this.

[0078] In the disclosed embodiment, the influence on misidentification is considered from the aspect of image quality to reduce the probability of misidentification caused by the influence of image quality, thereby ensuring the accuracy of production safety detection of the target object.

[0079] In some embodiments, when the safety detection result for the target object in the second image captured by the multiple second image sensors indicates that there is a production safety hazard and the credibility of the detection result is greater than or equal to a credibility threshold, safety hazard warning information for the target object is generated.

[0080] Among them, when it is greater than or equal to the credibility threshold, in this case, if the computer vision model determines that there is a safety hazard, there is a high probability that there is a safety hazard, and a safety hazard warning information can be directly generated.

[0081] In the disclosed embodiment, when most image sensors determine that a security risk exists for the same target object, there is no need to match with a whitelist, and security risk warning information can be directly generated to reduce the consumption of computing resources.

[0082] In some embodiments, the object feature is matched with the reference feature of the misdetected object in the whitelist to obtain a matching result, such as Figure 4 As shown, it can be implemented as:

[0083] S401, obtaining a device identification of an image sensor for capturing a target image and position information of the image sensor.

[0084] Among them, the posture information can be understood as the position and posture of the image sensor.

[0085] S402: Based on the device identification and the position information, search the cache for a first candidate set included in the whitelist.

[0086] In addition to the aforementioned equipment identification and posture information, filtering can also be performed in combination with labels, where labels are used to indicate the categories of production safety hazards.

[0087] The cache may be eliminated using LFU (Least Frequently Used) or other methods, which are not limited in the embodiments of the present disclosure.

[0088] S403: Determine the similarity between the object feature and the reference feature of at least one misdetected object in the first candidate set.

[0089] The cosine similarity may be used to calculate the similarity between the object feature and the reference feature of at least one misdetected object in the first candidate set, or other methods may be used to calculate the similarity between the two, which is not limited in the embodiments of the present disclosure.

[0090] S404: When the similarity between the reference feature and the object feature of the misdetected object in the first candidate set is greater than a preset threshold, a matching result is obtained that the object feature and the reference feature are successfully matched.

[0091] In some embodiments, in response to an update operation on the preset threshold, the preset threshold is updated; wherein an initial value of the preset threshold is determined based on the computer vision model.

[0092] Among them, the initial value of the preset threshold can be a confidence threshold used by the computer vision model to determine whether there are production safety hazards.

[0093] During implementation, the preset threshold can be adjusted based on the visual interface. The visual interface is shown in the figure below. Figure 5 As shown, in a more stringent scenario, the preset threshold can be adjusted to a higher threshold, and in a more relaxed scenario, the preset threshold can be adjusted to a lower threshold.

[0094] The visual interface can also be used to manage each production safety hazard, such as Figure 5 As shown, the interface includes detection of dangerous behaviors and behaviors of not wearing a helmet. If there is no need to detect dangerous behaviors, the detection function can be turned off.

[0095] In the disclosed embodiment, the user is supported to flexibly set the threshold according to the needs, so as to adjust the sensitivity of the whitelist filtering strategy and quickly adapt to different scenario requirements.

[0096] When the similarity between the reference feature of the falsely detected object in the first candidate set and the object feature is not greater than a preset threshold, the matching result is that the object feature successfully matches the reference feature or fails to match.

[0097] Among them, matching failure means that the object feature does not match the reference feature.

[0098] Because the data is frequently read from the database, the speed is slow, and most of the data is duplicated, so the cache is used to increase the speed of loading data, thereby speeding up the entire reasoning. During implementation, the lazy loading mode is used to achieve the matching process with the whitelist. The lazy loading mode is to obtain the reference features from the database in the case of the first request, and then asynchronously put them into the cache. The next time the request is received, the reference features are directly obtained from the cache, which can greatly reduce external read and write operations and increase the speed of loading the whitelist for matching.

[0099] In the disclosed embodiment, based on the device identification and the posture information, a search is first performed in the cache, and then based on a similarity matching method, a matching result is determined. Using the cache search method, reference features can be quickly located to ensure the efficiency of the entire security detection.

[0100] In some embodiments, if no reference feature is matched in the cache, Figure 6 As shown, it can be implemented as:

[0101] S601: When no reference feature is matched in the first candidate set, a second candidate set included in the whitelist is searched in the database based on the device identification and posture information.

[0102] The database is a database of reference features including all falsely detected objects in the whitelist.

[0103] S602: Determine the similarity between the object feature and the object feature of at least one misdetected object in the second candidate set.

[0104] The method of calculating the similarity has been described above and will not be repeated here in the embodiment of the present disclosure.

[0105] S603, when there is a reference feature of a falsely detected object in the second candidate set and the similarity between the feature of the object is greater than a preset threshold, a matching result is obtained that the feature of the object matches the reference feature successfully.

[0106] In the case of a successful match, that is, in the case of a match on the reference feature, the reference feature is updated to the cache for subsequent use.

[0107] When the similarity between the reference feature of the falsely detected object in the second candidate set and the object feature is not greater than a preset threshold, the matching result is a matching failure, that is, the reference feature is not matched, and a safety hazard alarm message is generated to inform relevant personnel to take corresponding measures.

[0108] In the disclosed embodiment, a multi-layer search method is used. If the search is not found in the cache, the search is performed in the database. At the same time, the search is first performed based on the device identification and the posture information, and then the match is performed based on the similarity. This can improve the accuracy of the matching whitelist to a certain extent.

[0109] In some embodiments, in order to implement the matching process more quickly, the object features and the reference features of the misdetected objects in the whitelist are matched to obtain a matching result, which can be specifically implemented as follows: constructing an index library corresponding to the reference features in the whitelist; calculating the similarity between the object features and the index library to obtain the matching result.

[0110] Among them, the index library corresponding to the reference features in the whitelist can be constructed through IndexFlatL2 (brute force indexing), and then the similarity is calculated using the cosine similarity method to obtain the matching result.

[0111] During implementation, matching is performed for reference features in the cache, which can be implemented as follows: based on the device identification and the posture information, searching the cache for the first candidate set included in the whitelist; filtering the index library corresponding to the first candidate set from the index library to construct a candidate index library; calculating the similarity between the object features and the candidate index library to obtain a matching result.

[0112] In addition, the index library may also be constructed by using IndexIVFFlat (inverted index) and IndexIVFPQ (space-saving index), which is not limited in the embodiments of the present disclosure.

[0113] In the disclosed embodiment, an efficient index library is used to perform similarity calculation, which further improves the matching speed and ensures the matching efficiency of the whitelist.

[0114] In some embodiments, when it is determined that the computer vision model generates a new falsely detected object, in response to a request to update the whitelist, a reference feature extracted by the computer vision model for the new falsely detected object is obtained; and the reference feature is added to the whitelist.

[0115] During implementation, if the matching result of the target image is a matching failure, that is, it does not match the reference features, a safety hazard alarm message is generated to inform relevant personnel to take corresponding measures. If the relevant personnel determine that the target image is a false detection object, it is selected as a false detection object and added to the whitelist.

[0116] When implementing, Figure 7 As shown, when the application platform receives a request from relevant personnel to add a falsely detected object to a whitelist, the database stores the falsely detected object in the database and sends a response to the application platform to inform that the false detection result has been stored in the database. The application platform then notifies the relevant personnel that the storage has been completed. At the same time, the model post-processing device, that is, the device that executes the image processing method provided by the embodiment of the present disclosure, can receive an inference request, which is a request generated based on the result of a safety detection based on a computer vision model. The request includes information about the target object in the target image, mainly including object features, production safety hazard category labels, image sensor device identification, and posture information. The model post-processing device can continuously process the inference request and match it with the whitelist, and the whitelist matching operation can be completed through the lazy loading described above. When the reference feature is not matched in the cache but is matched in the database, the reference feature is synchronized to the cache.

[0117] In some embodiments, there may be a large number of reference features in the same tag of the whitelist database. In this case, cluster analysis of the reference features can be performed to obtain different class centers, each of which corresponds to a reference feature. Based on this approach, the number of reference features in the whitelist can be reduced, and the whitelist matching efficiency can be improved.

[0118] During implementation, the Mini Batch K-Means algorithm can be used to perform cluster analysis on the reference features to obtain multiple target objects. During specific implementation, some data are randomly extracted from multiple reference features as a sample set, and K cluster points are constructed using the K-Means algorithm (clustering algorithm); then, some data are randomly extracted from the remaining data of the reference features except the sample set as a new sample set, and the K cluster points are trained based on the new sample set to update the center point of the cluster; the above operation is performed repeatedly until the center point is stable or the number of iterations is reached, and the calculation operation is stopped to obtain K cluster points, which are K reference features, where K is a positive integer.

[0119] In the disclosed embodiment, the user is supported to update the whitelist in real time and flexibly, so as to ensure the real-time nature of the entire security detection and ensure the efficiency of the entire security detection at a relatively low cost.

[0120] In some embodiments, in order to save computing resources of the computer vision model, it can also be implemented as follows: performing anomaly detection on the image state of the image to be processed to obtain anomaly detection results; when the anomaly detection indicates that there is no abnormality in the image to be processed, inputting the image to be processed into the computer vision model, and the computer vision model performs production safety hazard detection on the target image.

[0121] During implementation, advance detection and identification are performed for abnormal situations (such as occlusion, distant human bodies, rainy days, and foggy days) to obtain abnormal detection results. When the abnormal detection indicates that the image to be processed is abnormal, it is discarded, that is, the computer vision model is not used to process the image.

[0122] In the disclosed embodiment, the image state is identified in advance to reduce the computational process of normal reasoning of the computer vision model, reduce the computational process of processing using the computer vision model package and the whitelist, and save computing resources.

[0123] In the embodiment of the present disclosure, the system architecture diagram of the proposed image processing method is as follows: Figure 8As shown, the user performs user operations on the application platform, obtains a video stream based on an image sensor, performs frame cutting processing on the video stream based on a video service, and obtains a picture. The picture is input into a computer vision model to obtain a determination result of production safety detection. When the determination result indicates that there are production safety hazards in the target image, and the offset between the detection confidence of the candidate object in the target image and the confidence threshold is less than the offset threshold, the candidate object is taken as the target object and post-processing is performed. That is, the object features extracted by the computer vision model for the target object are extracted; the object features are matched with the reference features of the misdetected objects in the whitelist to obtain a matching result; when the object features match the reference features, the detection result of the computer vision model for the target object is corrected to the target object does not have a production safety hazard, that is, no alarm processing is performed on it; when the object features do not match the reference features, a safety hazard alarm information is generated to inform relevant personnel to perform corresponding processing.

[0124] Based on the same technical concept, the present disclosure also proposes an image processing device 900, such as Fig. 9 As shown, including:

[0125] An extraction module 901 is used to extract object features extracted by the computer vision model for the target object when the computer vision model determines that the target object in the target image has a production safety hazard;

[0126] A first matching module 902 is used to match the object features with reference features of the misdetected objects in the whitelist to obtain a matching result;

[0127] The correction module 903 is used to correct the detection result of the computer vision model for the target object to indicate that there is no production safety hazard for the target object when the object feature matches the reference feature.

[0128] In some embodiments, the system further includes a first processing module for:

[0129] When the computer vision model determines that multiple candidate objects have production safety hazards, obtaining detection confidences generated by the computer vision model for the multiple candidate objects;

[0130] Based on the detection confidences of the multiple candidate objects, the candidate objects whose detection confidences have an offset from a confidence threshold that is less than an offset threshold are screened out as the target objects.

[0131] In some embodiments, the system further includes a second processing module for:

[0132] For the candidate objects among the multiple candidate objects, for which the offset between the detection confidence and the confidence threshold is greater than or equal to the offset threshold, production safety hazard warning information for the candidate objects is generated.

[0133] In some embodiments, when the target object is a mobile target, the target image is an image in an image group, the image group is a safety detection performed by the computer vision model on the mobile target, and the detection result of the safety detection is that the motion trajectory and / or motion state of the mobile target meets the safety hazard alarm requirements.

[0134] In some embodiments, a third processing module is further included, configured to:

[0135] In the case where at least one image in the image group matches the reference feature, an alarm message generated for the image group is filtered out.

[0136] In some embodiments, a fourth processing module is further included, configured to:

[0137] Tracking and analyzing the target object based on the image to be processed acquired by at least one first image sensor to obtain state features of the target object in multiple frames of the first image;

[0138] Matching the state features of the target object in the multiple frames of first images with the potential safety hazard warning requirements;

[0139] In a case where the state characteristics of the target object in the multiple frames of first images meet the safety hazard warning requirements, the image group is constructed based on the multiple frames of first images.

[0140] In some embodiments, a fifth processing module is further included, configured to:

[0141] Capturing images of the safety area based on a plurality of second image sensors;

[0142] Performing safety detection on the target object in the second images captured by the plurality of second image sensors based on the computer vision model;

[0143] Based on the fact that the safety detection result for the target object in the second images captured by the multiple second image sensors is that there is a production safety hazard and the credibility of the detection result is less than a credibility threshold, the target image is filtered out from the second images captured by the multiple second image sensors.

[0144] In some embodiments, the fifth processing module is specifically configured to:

[0145] Performing quality evaluation on the image portion of the target object in the second images captured by the plurality of second image sensors to obtain image quality;

[0146] A second image whose image quality meets a preset requirement is selected as the target image.

[0147] In some embodiments, a sixth processing module is further included, configured to:

[0148] If a safety detection result for the target object in the second image captured by the multiple second image sensors indicates that there is a production safety hazard and the credibility of the detection result is greater than or equal to a credibility threshold, safety hazard warning information for the target object is generated.

[0149] In some embodiments, an additional module is further included for:

[0150] In a case where it is determined that the computer vision model generates a new falsely detected object, in response to a request to update the whitelist, obtaining a reference feature extracted by the computer vision model for the new falsely detected object;

[0151] The reference signature is added to the whitelist.

[0152] In some embodiments, the first matching module includes:

[0153] An acquisition unit, used to acquire a device identification of an image sensor that acquires the target image and position information of the image sensor;

[0154] A searching unit, configured to search a cache for a first candidate set included in the whitelist based on the device identifier and the posture information;

[0155] a first computing unit, configured to determine a similarity between the object feature and a reference feature of at least one misdetected object in the first candidate set;

[0156] The matching unit is used to obtain the matching result that the object feature successfully matches the reference feature when the similarity between the reference feature of the falsely detected object in the first candidate set and the object feature is greater than a preset threshold.

[0157] In some embodiments, a second matching module is further included, which is used to:

[0158] When no reference feature is matched in the first candidate set, searching a database for a second candidate set included in the whitelist based on the device identifier and the posture information;

[0159] determining a similarity between the object feature and an object feature of at least one misdetected object in the second candidate set;

[0160] In the case where the similarity between the reference feature of the misdetected object in the second candidate set and the object feature is greater than a preset threshold, the matching result is that the object feature and the reference feature are successfully matched.

[0161] In some embodiments, an update module is further included, which is used to:

[0162] In response to an update operation on the preset threshold, updating the preset threshold;

[0163] Wherein, the initial value of the preset threshold is determined based on the computer vision model.

[0164] In some embodiments, the first matching module includes:

[0165] A construction unit, used to construct an index library corresponding to the reference features in the whitelist;

[0166] The second calculation unit is used to calculate the similarity between the object feature and the index library to obtain the matching result.

[0167] In some embodiments, an anomaly detection module is further included, which is used to:

[0168] Perform anomaly detection on the image state of the image to be processed to obtain an anomaly detection result;

[0169] When the abnormality detection result indicates that the image to be processed does not have a preset abnormality, the image to be processed is input into the computer vision model, and the computer vision model performs production safety hazard detection on the target image.

[0170] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0171] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0172] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0173] Fig.10A schematic block diagram of an example electronic device 1000 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0174] like Fig.10 As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0175] A number of components in the device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0176] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as image processing methods. For example, in some embodiments, the image processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the image processing method described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to perform the image processing method in any other appropriate manner (e.g., by means of firmware).

[0177] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0178] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0179] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0180] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0181] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0182] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0183] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0184] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An image processing method, comprising: When the computer vision model determines that a target object in the target image has a production safety hazard, extracting object features extracted by the computer vision model for the target object; Performing a matching operation on the object features and the reference features of the misdetected objects in the whitelist to obtain a matching result; In the case where the object feature matches the reference feature, the detection result of the computer vision model for the target object is corrected to indicate that there is no production safety hazard for the target object.

2. The method according to claim 1, wherein: Determining the target object includes: When the computer vision model determines that multiple candidate objects have production safety hazards, obtaining detection confidences generated by the computer vision model for the multiple candidate objects; Based on the detection confidences of the multiple candidate objects, the candidate objects whose detection confidences have an offset from a confidence threshold that is less than an offset threshold are screened out as the target objects.

3. The method according to claim 2, further comprising: For the candidate objects among the multiple candidate objects, for which the offset between the detection confidence and the confidence threshold is greater than or equal to the offset threshold, production safety hazard warning information for the candidate objects is generated.

4. The method according to claim 1, wherein: In the case where the target object is a mobile target, the target image is an image in an image group, the image group is a safety detection performed by the computer vision model on the mobile target, and the detection result of the safety detection is that the motion trajectory and / or motion state of the mobile target meets the safety hazard alarm requirements.

5. The method according to claim 4, further comprising: In the case where at least one image in the image group matches the reference feature, an alarm message generated for the image group is filtered out.

6. The method according to claim 4, wherein: Determining the image group includes: Tracking and analyzing the target object based on the image to be processed acquired by at least one first image sensor to obtain state features of the target object in multiple frames of the first image; Matching the state features of the target object in the multiple frames of first images with the potential safety hazard warning requirements; In a case where the state characteristics of the target object in the multiple frames of first images meet the safety hazard warning requirements, the image group is constructed based on the multiple frames of first images.

7. The method according to claim 1, wherein: Determining the target image includes: Capturing images of the safety area based on a plurality of second image sensors; Performing safety detection on the target object in the second images captured by the plurality of second image sensors based on the computer vision model; Based on the fact that the safety detection result for the target object in the second images captured by the multiple second image sensors is that there is a production safety hazard and the credibility of the detection result is less than a credibility threshold, the target image is filtered out from the second images captured by the multiple second image sensors.

8. The method according to claim 7, wherein: The step of screening out the target image from the second images acquired by the plurality of second image sensors comprises: Performing a quality evaluation on an image portion of the target object in the second images captured by the plurality of second image sensors to obtain image quality; A second image whose image quality meets a preset requirement is selected as the target image.

9. The method according to claim 7, further comprising: If a safety detection result for the target object in the second image captured by the multiple second image sensors indicates that there is a production safety hazard and the credibility of the detection result is greater than or equal to a credibility threshold, safety hazard warning information for the target object is generated.

10. The method according to any one of claims 1 to 9, further comprising: In a case where it is determined that the computer vision model generates a new falsely detected object, in response to a request to update the whitelist, obtaining a reference feature extracted by the computer vision model for the new falsely detected object; The reference signature is added to the whitelist.

11. The method according to claim 1, wherein: The matching operation of the object feature with the reference feature of the misdetected object in the whitelist to obtain a matching result includes: Acquire a device identification of an image sensor that acquires the target image and position information of the image sensor; Based on the device identification and the posture information, searching the cache for a first candidate set included in the whitelist; determining a similarity between the object feature and a reference feature of at least one misdetected object in the first candidate set; In the case where the similarity between the reference feature of the misdetected object in the first candidate set and the object feature is greater than a preset threshold, the matching result is that the object feature successfully matches the reference feature.

12. The method according to claim 11, further comprising: When no reference feature is matched in the first candidate set, searching a database for a second candidate set included in the whitelist based on the device identifier and the posture information; determining a similarity between the object feature and an object feature of at least one misdetected object in the second candidate set; In the case where the similarity between the reference feature of the misdetected object in the second candidate set and the object feature is greater than a preset threshold, the matching result is that the object feature successfully matches the reference feature.

13. The method according to claim 11 or 12, further comprising: In response to an update operation on the preset threshold, updating the preset threshold; Wherein, the initial value of the preset threshold is determined based on the computer vision model.

14. The method according to any one of claims 1 to 13, wherein: The matching operation of the object feature with the reference feature of the misdetected object in the whitelist to obtain a matching result includes: Constructing an index library corresponding to the reference features in the whitelist; The similarity between the object feature and the index library is calculated to obtain the matching result.

15. The method according to any one of claims 1 to 14, further comprising: Perform anomaly detection on the image state of the image to be processed to obtain an anomaly detection result; When the abnormality detection result indicates that the image to be processed does not have a preset abnormality, the image to be processed is input into the computer vision model, and the computer vision model performs production safety hazard detection on the target image.

16. An image processing device, comprising: An extraction module, for extracting object features extracted by the computer vision model for the target object when the computer vision model determines that the target object in the target image has a production safety hazard; A first matching module is used to match the object features with reference features of misdetected objects in the whitelist to obtain a matching result; A correction module is used to correct the detection result of the computer vision model for the target object to indicate that there is no production safety hazard for the target object when the object feature matches the reference feature.

17. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 15.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-15.

19. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 15.

Images