Image processing methods, apparatus, devices and storage media

Abstract

This disclosure provides image processing methods, apparatus, devices, and storage media, relating to the field of image processing technology, and particularly to the fields of artificial intelligence, computer vision, big data, and deep learning. The specific implementation scheme is as follows: when a computer vision model determines that a target object in a target image poses a production safety hazard, the object features extracted by the computer vision model for the target object are extracted; the object features are matched with reference features of falsely detected objects in a whitelist to obtain a matching result; if the object features match the reference features, the detection result of the computer vision model for the target object is corrected to indicate that the target object does not pose a production safety hazard.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to the fields of artificial intelligence, computer vision, big data, and deep learning. Background Technology

[0002] With the continuous development of artificial intelligence, computer vision (CV) model reasoning and recognition technology has been widely applied in various fields. Current CV models are basically data-driven and can analyze and process image data to obtain the desired results. Summary of the Invention

[0003] This disclosure provides an image processing method, apparatus, device, and storage medium.

[0004] According to one aspect of this disclosure, an image processing method is provided, comprising:

[0005] When a computer vision model determines that a target object in a target image poses a production safety hazard, the object features extracted by the computer vision model for the target object are extracted.

[0006] The object features are matched with the reference features of falsely detected objects in the whitelist to obtain the matching results;

[0007] If the object features match the reference features, the detection result of the computer vision model for the target object is corrected to indicate that the target object does not pose a production safety hazard.

[0008] According to another aspect of this disclosure, an image processing apparatus is provided, comprising:

[0009] The extraction module is used to extract the object features extracted by the computer vision model for the target object when the computer vision model determines that the target object in the target image has a production safety hazard.

[0010] The first matching module is used to match the object features with the reference features of falsely detected objects in the whitelist to obtain the matching result;

[0011] The correction module is used to correct the detection result of the computer vision model for the target object to the point that the target object does not pose a production safety hazard when the object features match the reference features.

[0012] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0013] At least one processor; and

[0014] The memory is communicatively connected to the at least one processor; wherein,

[0015] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform any of the methods described in the present disclosure.

[0016] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform any of the methods according to embodiments of this disclosure.

[0017] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the methods according to embodiments of this disclosure.

[0018] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0019] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0020] Figure 1 This is a schematic flowchart of an image processing method according to an embodiment of the present disclosure;

[0021] Figure 2 This is a schematic diagram of a target image according to an embodiment of the present disclosure;

[0022] Figure 3 This is an architecture diagram of an image processing method according to an embodiment of the present disclosure;

[0023] Figure 4 This is another schematic flowchart of an image processing method according to an embodiment of the present disclosure;

[0024] Figure 5 This is a schematic diagram of a visual interface according to an embodiment of the present disclosure;

[0025] Figure 6 This is another schematic flowchart of an image processing method according to an embodiment of the present disclosure;

[0026] Figure 7 This is a scene diagram of an image processing method according to an embodiment of the present disclosure;

[0027] Figure 8 This is a schematic diagram of the architecture of an image processing method according to an embodiment of the present disclosure;

[0028] Figure 9 This is a schematic diagram of the structure of an image processing apparatus according to an embodiment of the present disclosure;

[0029] Figure 10 This is a block diagram of an electronic device used to implement the image processing method of the embodiments of this disclosure. Detailed Implementation

[0030] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0031] The terms “first,” “second,” etc., used in this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion, such as including a series of steps or units. A method, system, product, or apparatus is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0032] In actual production environments, computer vision models can be used to inspect the production environment to ensure production safety. For example, to ensure the safety of workers in the production environment, their clothing can be inspected, and hazardous work areas can be detected.

[0033] However, production workshops are generally large and contain various fixed and mobile equipment. Some elements that need to be inspected may be obstructed. Furthermore, outdoor production environments can lead to poor image quality. In short, factors such as lighting or obstruction can cause computer vision models to misidentify objects. For example, a worker may be wearing a safety helmet, but due to obstruction or poor visibility, the computer vision model may identify them as not wearing one.

[0034] In related technologies, in order to reduce the false recognition rate of computer vision models, these false recognition data can be recycled and used to retrain the computer vision model. This allows for fine-tuning of model parameters and a reduction in the generation of false recognition data.

[0035] However, training computer vision models is a lengthy process that wastes a lot of manpower. Retraining also requires additional computing resources. These are not good solutions for R&D teams. Moreover, retraining does not necessarily achieve the expected generalization effect, and the inference performance may not improve, so the return on investment needs to be improved.

[0036] In view of this, an image processing method is proposed in this embodiment. In this embodiment, considering that computer vision models may identify incorrect results during inference due to scene issues or boundary conditions, users can manually add these incorrectly identified results to a set to form a whitelist on the application platform. Using the whitelist in model post-processing can filter out data misidentified by the computer vision model, thereby improving the accuracy of alarms.

[0037] like Figure 1 The diagram shown is a flowchart illustrating an image processing method provided in this embodiment of the present disclosure, including the following:

[0038] S101, when the computer vision model determines that the target object in the target image poses a production safety hazard, extract the object features extracted by the computer vision model for the target object.

[0039] The target image can be any image from an actual production environment. For example, regarding the target object, such as... Figure 2 As shown, the target object can be a human body in each identification label box. Alternatively, the target object can be equipment in a production environment, such as metering pumps or air pressure devices; however, this disclosure does not limit the target object.

[0040] For human subjects, production safety hazards can include situations such as not wearing a safety helmet or performing dangerous operations.

[0041] S102, perform a matching operation between the object features and the reference features of the falsely detected objects in the whitelist to obtain the matching result.

[0042] S103, when the object features match the reference features, the detection result of the computer vision model for the target object is corrected to indicate that the target object does not pose a production safety hazard.

[0043] Even if the object features do not match the reference features, it can be confirmed that the production safety hazard identified by the computer vision model is correct, and subsequent processes can be executed.

[0044] In this embodiment, a whitelist is used to assist the computer vision model. Based on matching, misidentifications can be quickly identified and corrected. Since in real-world production environments, computer vision models are typically input as video streams, the whitelist-based matching method allows for rapid filtering of misidentified results, reducing unnecessary alarms. The whitelist helps improve the final alarm efficiency of the computer vision model. The entire process does not require data retrieval or model retraining, and can be quickly applied to production environments, improving alarm accuracy. In summary, the processing results of the computer vision model can be corrected at a relatively low cost, reducing the false alarm rate.

[0045] Especially during the iterative development of a project, post-processing with a whitelist can better safeguard the computer vision model and ensure its successful implementation. This method is highly efficient, easy to implement, and yields controllable results. It also saves resources needed for training the model and does not negatively impact the computer vision model or other modules.

[0046] Exemplary architecture diagrams applicable to embodiments of this disclosure are as follows: Figure 3 As shown, the system includes a computer vision model, a model package, and a whitelist. The image to be processed is input into the computer vision model to obtain the image's data structure. This data structure includes, but is not limited to, the size of the label box, the coordinates of its top-left corner, the object features of each identified target object, and detection confidence. The model package processes the target image's data structure to obtain the coordinates of the identified label boxes, labels, and object features. The model package can be used to visualize the output of the computer vision model, simultaneously displaying specific types of production safety hazards.

[0047] When a computer vision model determines that an image to be processed poses a production safety hazard, the image to be processed is designated as the target image, and the target objects within the target image that pose a production safety hazard are identified. Subsequently, these target objects are matched against a whitelist, and further processing is performed based on the matching results. Therefore, by reviewing the whitelist, the accuracy of alarms can be improved.

[0048] The model package can be one or more. When there is only one model package, it can include multiple tags to correspond to various production safety hazards. For example, in detecting potential fire hazards, the tags can include open flame, smoke, and human body. When there are multiple model packages, each production safety hazard can correspond to one model package, or multiple production safety hazards can correspond to one model package. For example, if the computer vision model frequently misidentifies a certain production safety hazard 1, this type of production safety hazard can be assigned to one model package. If production safety hazards 2 and 3 are misidentified less frequently, they can be assigned to one model package, with one model package handling post-processing for both hazards to save computational resources. It is understood that this disclosure does not limit the scope of the embodiments.

[0049] In some embodiments, such as Figure 2 As shown, a single target image may contain multiple objects, each of which is detected by a computer vision model to determine if there is a safety hazard. Therefore, in this embodiment, determining the target object can be implemented as follows: when the computer vision model determines that multiple candidate objects pose a production safety hazard, the detection confidence level generated by the computer vision model for the multiple candidate objects is obtained; based on the detection confidence levels of the multiple candidate objects, candidate objects whose detection confidence level deviates from a confidence level threshold by less than a threshold value are selected as the target object.

[0050] In practice, the confidence threshold is set to 60% and the offset threshold is set to 30%. The detection confidence of candidate object 1 is 65%, the detection confidence of candidate object 2 is 78%, and the detection confidence of candidate object 3 is 98%. In this case, the offset of candidate object 1 is 5% and the offset of candidate object 2 is 18%. Therefore, candidate object 1 and candidate object 2 are target objects and need to be matched through the subsequent whitelist.

[0051] In this embodiment, the offset between the detection confidence level and the confidence threshold can be used to determine which candidate objects are the target objects and require subsequent re-examination and filtering based on a whitelist. Therefore, by filtering out the target objects that require post-processing, the number of queries and whitelist matching operations is reduced. This approach can save computational resources to a certain extent while improving re-examination efficiency.

[0052] In some embodiments, among the plurality of candidate objects, candidate objects whose confidence level is offset from the confidence level threshold is greater than or equal to the offset threshold are detected, and production safety hazard alarm information is generated for the candidate objects.

[0053] Continuing with the previous example, if the offset of candidate object 3 is 38%, it is greater than the offset threshold. In this case, the probability that candidate object 3 has unsafe factors is relatively high. Therefore, there is no need to re-check through the whitelist. Security risk alarm information can be generated directly to notify relevant personnel to check and deal with security risks in a timely manner.

[0054] In this embodiment of the disclosure, since the probability of an object having a security risk is relatively high when the offset between the detection confidence level and the confidence level threshold is greater than or equal to the offset threshold, there is no need to perform whitelist detection and alarm information can be generated directly. This can reduce the computing resources required for whitelist matching to a certain extent and improve alarm efficiency.

[0055] In some embodiments, the target object may be a moving target in the actual production environment, requiring it to move within the environment. During actual production, to improve the accuracy of alarms and detect safety hazards in moving targets, it is necessary to track the movement of the target, typically using multiple frames of images to determine if a safety hazard exists. Therefore, when the target object is a moving target, the target image is an image from an image group, where the image group represents the computer vision model performing safety detection on the moving target, and the detection result indicates that the moving target's trajectory and / or motion state meet the safety hazard alarm requirements.

[0056] Specifically, when the movement trajectory indicates that the movement has crossed the boundary or deviated from the original compliant movement route, it is determined that the safety hazard alarm requirements are met; when the movement status indicates the state of the target object, there may be safety hazards, such as a person falling down or not wearing a safety helmet.

[0057] In this embodiment of the disclosure, since the target object is in motion, the movement process of the target object needs to be organized into an image group, and security detection is performed on the target object in the group to ensure the accuracy of moving target detection, so as to avoid the problem of misidentification caused by partial detection and improve the overall efficiency of security detection.

[0058] In some embodiments, determining an image group for a moving target can be implemented as follows: tracking and analyzing the target object based on images to be processed acquired by at least one first image sensor to obtain the state features of the target object in multiple first images; matching the state features of the target object in the multiple first images with security hazard alarm requirements; and constructing the image group based on the multiple first images if the state features of the target object in the multiple first images meet the security hazard alarm requirements.

[0059] In implementation, the ID (IdentityDocument) of the same target object in a preset number of frames of images to be processed can be set to the same ID. The state features of the target object are extracted from each frame. For example, the state features of the target object in three consecutive frames can be extracted using a computer vision model. These three frames are used as the first images to construct an image group. The state features of the target object in these three first images are matched against safety hazard alarm requirements. If the state features in the three consecutive first images meet the safety hazard alarm requirements, it is determined that the target object poses a production safety hazard, facilitating subsequent whitelist matching operations.

[0060] In this embodiment of the disclosure, the target object is tracked so that the state characteristics of the target object form an overall image group, which facilitates subsequent re-examination of the entire image group using a whitelist, thereby improving the alarm accuracy for moving targets.

[0061] In some embodiments, if at least one image in the image group matches a reference feature, alarm messages generated for the image group are filtered out.

[0062] In implementation, continuing from the previous example, after constructing the three first images into an image group corresponding to the target object, a matching operation is performed with the reference features of the falsely detected objects in the whitelist to obtain the matching result; if the reference feature is matched in any of the three frames, the alarm message generated for the image group is filtered out; if the reference feature is not matched in any of the three frames, an alarm message is generated for the image group.

[0063] In this embodiment of the disclosure, a whitelist can be used to improve the accuracy of alarms for moving targets, thereby reducing unnecessary alarms.

[0064] In some embodiments, in order to further improve post-processing efficiency, the target image that needs to be re-inspected can be determined as follows: if the safety detection result of the target object in the second image acquired by the plurality of second image sensors is that there is a production safety hazard and the credibility of the detection result is less than the credibility threshold, the target image is selected from the second image acquired by the plurality of second image sensors.

[0065] The safe zone can be set based on the actual situation, and this disclosure does not limit it.

[0066] During implementation, due to the angle and position of the second image sensor, or factors such as lighting and weather, the image quality may be low, leading to some target objects being identified as having production safety hazards. In such cases, the second images containing safety hazards can be selected as target images for subsequent matching with the whitelist. Whether to select target images based on confidence level depends on the angle and / or lighting and weather conditions of the second image sensor at the time. For example, if images captured by the second image sensor at a certain angle are mostly obstructed, confidence level-based target image selection can be initiated. Similarly, if the second image sensor in an outdoor environment captures poor-quality images during thunderstorms or heavy fog, confidence level-based target object selection can be initiated. Correspondingly, it is understandable that the whitelist also contains reference characteristics of falsely detected objects under such circumstances.

[0067] In this embodiment of the disclosure, considering the credibility of the second image, that is, taking into account the quality of the second image, if the credibility is low, the target image is then selected for subsequent whitelist matching, which can reduce unnecessary resource consumption to a certain extent.

[0068] In some embodiments, determining the credibility of the security detection results for the target object can be implemented as follows:

[0069] Images of the safe area are acquired using multiple second image sensors; based on the importance of the multiple second image sensors, corresponding weights are assigned to the second images acquired by each second image sensor; based on the corresponding weights assigned to the second images acquired by each second image sensor, the detection results of the computer vision model for production safety detection of the target object in the second images acquired by the multiple second image sensors are weighted and summed to obtain the credibility of the safety detection results of the target object.

[0070] For example, in a scenario, there might be a primary second image sensor and two secondary image sensors. The weight of the second image acquired by the primary second image sensor can be set to 60%, and the weight of the second images acquired by the two secondary second image sensors can be set to 20%. If the detection result of the second image acquired by the primary second image sensor is 80% of the probability that there is a production safety hazard, and the detection results of the second images acquired by the two secondary second image sensors are 75% and 72% respectively, then the confidence level of the safety detection result of the target object is (60%*80+20%*75+20%*72)=77.4. This confidence level is then compared with a confidence threshold, and subsequent processing is performed.

[0071] In this embodiment, considering that the importance of each second image sensor is not the same, corresponding weights can be assigned to them to improve the accuracy of security detection of images. The specific weights can be set based on actual circumstances, and this embodiment does not limit this.

[0072] In another embodiment, determining the credibility of the security detection results of the target object can also be implemented by: performing security detection on the target object in the second images acquired by the plurality of second image sensors based on the computer vision model; and determining the proportion of second image sensors that are judged to have production safety hazards as the credibility.

[0073] For example, in the case of 5 second image sensors, if the second images acquired by 3 second image sensors all indicate that the target object has a safety hazard, then the proportion is 60%. If the second images acquired by 2 second image sensors are determined to have a production safety hazard, the confidence level is 40%. Then, the confidence level threshold is used to filter out the target image from the second images acquired by the multiple second image sensors if the value is below the confidence level threshold.

[0074] In some embodiments, the step of selecting the target image from the second images acquired by the plurality of second image sensors may be implemented as follows: performing quality evaluation on the image portion of the target object in the second images acquired by the plurality of second image sensors to obtain image quality; and selecting second images whose image quality meets preset requirements as the target image.

[0075] The preset requirement can be a score lower than the preset quality score.

[0076] During implementation, an image quality assessment model can be used to evaluate the quality of the image and output an image quality score. If the quality score is lower than the preset quality score, the image is used as the target image.

[0077] Any method that can determine image quality is applicable to the embodiments of this disclosure, and the embodiments of this disclosure are not limited thereto.

[0078] In this embodiment of the disclosure, the impact of image quality on misidentification is considered to reduce the probability of misidentification caused by image quality issues, thereby ensuring the accuracy of production safety detection of the target object.

[0079] In some embodiments, if the safety detection result of the target object in the second image acquired by the plurality of second image sensors indicates that there is a production safety hazard and the credibility of the detection result is greater than or equal to the credibility threshold, a safety hazard alarm message for the target object is generated.

[0080] If the confidence threshold is greater than or equal to the threshold, and the computer vision model determines that there is a security risk, then there is a high probability that there is a security risk, and a security risk alarm message can be generated directly.

[0081] In this embodiment of the disclosure, when most image sensors determine that there is a security risk for the same target object, there is no need to match with a whitelist, and security risk alarm information can be directly generated to reduce the consumption of computing resources.

[0082] In some embodiments, the step of matching the object features with the reference features of falsely detected objects in the whitelist to obtain a matching result is as follows: Figure 4 As shown, it can be implemented as follows:

[0083] S401, acquire the device identifier of the image sensor that acquires the target image and the pose information of the image sensor.

[0084] Pose information can be understood as the position and orientation of the image sensor.

[0085] S402, based on device identifier and pose information, searches the cache for the first candidate set contained in the whitelist.

[0086] In addition to the aforementioned equipment identification and position information, labels can also be used for filtering, where labels are used to indicate the category of production safety hazards.

[0087] The cache can be evicted using LFU (Least Frequently Used) or other methods, and this disclosure does not limit the specific methods used.

[0088] S403, determine the similarity between the object features and the reference features of at least one falsely detected object in the first candidate set.

[0089] The similarity between object features and reference features of at least one falsely detected object in the first candidate set can be calculated using cosine similarity, or other methods can be used to calculate the similarity between the two. This disclosure does not limit the specific methods used.

[0090] S404, if the similarity between the reference feature and the object feature of a falsely detected object in the first candidate set is greater than a preset threshold, the matching result is that the object feature and the reference feature are successfully matched.

[0091] In some embodiments, the preset threshold is updated in response to an update operation on the preset threshold; wherein the initial value of the preset threshold is determined based on the computer vision model.

[0092] The initial value of the preset threshold can be the confidence threshold used by the computer vision model to determine whether there are potential production safety hazards.

[0093] During implementation, the preset thresholds can be adjusted based on a visual interface, as shown in the image below. Figure 5 As shown, in more stringent scenarios, the preset threshold can be adjusted to a higher threshold, while in more lenient scenarios, the preset threshold can be adjusted to a lower threshold.

[0094] This visual interface can also be used to manage each type of production safety hazard, such as... Figure 5 As shown, this interface includes the detection of dangerous behaviors and not wearing a helmet. This detection function can be turned off when it is not necessary to detect dangerous behaviors.

[0095] In this embodiment of the disclosure, users can flexibly set thresholds according to their needs to adjust the sensitivity of the whitelist filtering strategy and quickly adapt to different scenario requirements.

[0096] If the similarity between the reference features of a falsely detected object in the first candidate set and the object features is not greater than a preset threshold, the matching result is either a successful match between the object features and the reference features or a failed match.

[0097] In this context, a failed match indicates that the object's features did not match the reference features.

[0098] Because frequent data reads from the database are slow and mostly consist of repetitive data, caching is used to improve data loading speed and thus accelerate the overall inference process. In implementation, a lazy loading pattern is used to handle the matching process with the whitelist. Lazy loading retrieves reference features from the database on the first request and asynchronously caches them. Subsequent requests retrieve the same features directly from the cache, significantly reducing external read / write operations and improving the speed of loading the whitelist for matching.

[0099] In this embodiment of the disclosure, based on the device identifier and the pose information, a search is first performed in the cache, and then a matching result is determined based on similarity matching. Using the cache search method, reference features can be quickly located, thus ensuring the efficiency of the entire security detection.

[0100] In some embodiments, if the reference feature is not matched in the cache, such as Figure 6 As shown, it can be implemented as follows:

[0101] S601, if no reference feature is matched in the first candidate set, search the database for the second candidate set included in the whitelist based on the device identifier and pose information.

[0102] The database is a reference feature database that includes all false positives from the whitelist.

[0103] S602, determine the similarity between object features and object features of at least one falsely detected object in the second candidate set.

[0104] The method for calculating similarity has been described above, and will not be repeated here in the embodiments disclosed herein.

[0105] S603, if the similarity between the reference feature and the object feature of a falsely detected object in the second candidate set is greater than a preset threshold, the matching result is that the object feature and the reference feature are successfully matched.

[0106] If a match is successful, i.e., if a reference feature is matched, the reference feature is updated in the cache for later use.

[0107] If the similarity between the reference features of a falsely detected object in the second candidate set and the object features is not greater than a preset threshold, the matching result is a failure, i.e., the reference features are not matched. In this case, a security risk alarm is generated to notify relevant personnel to take corresponding actions.

[0108] In this embodiment, a multi-level search method is used. If the item is not found in the cache, a search is then performed in the database. At the same time, the device identifier and the pose information are first filtered, and then the similarity is used for matching, which can improve the accuracy of the matching whitelist to a certain extent.

[0109] In some embodiments, to achieve a faster matching process, the object features are matched with the reference features of falsely detected objects in the whitelist to obtain a matching result. Specifically, this can be implemented by: constructing an index library corresponding to the reference features in the whitelist; and calculating the similarity between the object features and the index library to obtain the matching result.

[0110] One approach is to construct an index library corresponding to the reference features in the whitelist using IndexFlatL2 (brute force indexing), and then use cosine similarity to calculate the similarity and obtain the matching results.

[0111] In practice, matching based on reference features in the cache can be implemented as follows: based on the device identifier and the pose information, search for the first candidate set contained in the whitelist in the cache; filter the index library corresponding to the first candidate set from the index library to construct a candidate index library; calculate the similarity between the object features and the candidate index library to obtain the matching result.

[0112] In addition, an index library can also be built using IndexIVFFlat (inverted index) and IndexIVFPQ (space-saving index), but this disclosure does not limit this approach.

[0113] In this embodiment of the disclosure, a high-efficiency index library is used for similarity calculation, which further improves the matching speed and ensures the matching efficiency of the whitelist.

[0114] In some embodiments, when it is determined that the computer vision model generates a new false detection object, in response to a request to update the whitelist, reference features extracted by the computer vision model for the new false detection object are obtained; and the reference features are added to the whitelist.

[0115] During implementation, if the matching result of the target image is a failure, that is, it does not match the reference features mentioned above, a security risk alarm message will be generated to inform relevant personnel to take corresponding actions. If the relevant personnel determine that the target image is a false detection object, it will be selected as a false detection object and added to the whitelist.

[0116] When implementing, such as Figure 7 As shown, when the application platform receives a request from relevant personnel to add a falsely detected object to the whitelist, the database stores the falsely detected object and sends a response to the application platform, informing it that the false detection result has been stored. The application platform then notifies the relevant personnel that the storage is complete. Simultaneously, the model post-processing device, i.e., the device executing the image processing method provided in this embodiment, can receive inference requests. These inference requests are generated based on the results of security detection using a computer vision model. The requests include information about the target object in the target image, primarily including object features, production safety hazard category labels, image sensor device identifiers, and pose information. The model post-processing device can continuously process inference requests and match them with the whitelist. Whitelist matching can be completed through lazy loading as described above. If a reference feature is not matched in the cache but is matched in the database, the reference feature is synchronized to the cache.

[0117] In some embodiments, a large number of reference features may exist within the same tag in the whitelist database. In this case, cluster analysis of the reference features can be used to obtain different cluster centers, with each cluster center corresponding to one reference feature. This approach can reduce the number of reference features within the whitelist and improve whitelist matching efficiency.

[0118] In implementation, the Mini Batch K-Means algorithm can be used to perform cluster analysis on the reference features to obtain multiple target objects. Specifically, a portion of data is randomly selected from the multiple reference features to form a sample set, resulting in K cluster points constructed using the K-Means algorithm. Then, a portion of data is randomly selected from the remaining data of the reference features (excluding the sample set) to form a new sample set. The K cluster points are then trained based on this new sample set to update the cluster centroids. This process is repeated until the centroids stabilize or the required number of iterations is reached, at which point the computation stops, resulting in K cluster points. These K cluster points are the K reference features, where K is a positive integer.

[0119] In this embodiment of the disclosure, users are allowed to update the whitelist in real time and flexibly to ensure the real-time nature of the entire security detection and to ensure the efficiency of the entire security detection with less cost.

[0120] In some embodiments, in order to save the computing resources of the computer vision model, the method may be implemented as follows: performing anomaly detection on the image state of the image to be processed to obtain anomaly detection results; if the anomaly detection indicates that the image to be processed has no anomalies, inputting the image to be processed into the computer vision model, and having the computer vision model perform production safety hazard detection on the target image.

[0121] During implementation, abnormal situations (such as occlusion, distant human bodies, rainy days, and foggy days) are detected and identified in advance to obtain anomaly detection results. If the anomaly detection indicates that the image to be processed has an anomaly, it is discarded, that is, the image is not processed by computer vision model.

[0122] In this embodiment of the disclosure, the image state is identified in advance to reduce the computational process of normal inference of the computer vision model, reduce the computational process of using computer vision model packages and whitelists, and save computing resources.

[0123] The system architecture diagram of the proposed image processing method in this embodiment is as follows: Figure 8As shown, the user performs user operations on the application platform, acquires a video stream based on an image sensor, and performs frame-by-frame processing on the video stream based on a video service to obtain an image. The image is then input into a computer vision model to obtain a production safety detection judgment result. If the judgment result indicates that there is a production safety hazard in the target image, and the offset between the detection confidence score of a candidate object in the target image and the confidence score threshold is less than the offset threshold, the candidate object is taken as the target object and post-processing is performed. Specifically, the object features extracted by the computer vision model for the target object are extracted; the object features are matched with reference features of falsely detected objects in a whitelist to obtain a matching result; if the object features match the reference features, the detection result of the computer vision model for the target object is corrected to indicate that the target object does not pose a production safety hazard, i.e., no alarm is triggered; if the object features do not match the reference features, a safety hazard alarm message is generated to notify relevant personnel to take appropriate action.

[0124] Based on the same technical concept, this disclosure also proposes an image processing apparatus 900, such as... Figure 9 As shown, it includes:

[0125] Extraction module 901 is used to extract object features extracted by the computer vision model for the target object when the computer vision model determines that the target object in the target image has a production safety hazard;

[0126] The first matching module 902 is used to match the object features with the reference features of falsely detected objects in the whitelist to obtain a matching result;

[0127] The correction module 903 is used to correct the detection result of the computer vision model for the target object to the point that the target object does not pose a production safety hazard when the object features match the reference features.

[0128] In some embodiments, a first processing module is further included, configured to:

[0129] When the computer vision model determines that multiple candidate objects pose a production safety hazard, the detection confidence level generated by the computer vision model for the multiple candidate objects is obtained.

[0130] Based on the detection confidence of the multiple candidate objects, candidate objects whose detection confidence is less than the offset threshold are selected as the target objects.

[0131] In some embodiments, a second processing module is further included, for:

[0132] For the candidate objects among the multiple candidate objects, if the offset between the detection confidence level and the confidence threshold is greater than or equal to the offset threshold, a production safety hazard alarm message is generated for the candidate object.

[0133] In some embodiments, when the target object is a moving target, the target image is an image in an image group, the image group being the computer vision model performing security detection on the moving target, and the detection result of the security detection is that the motion trajectory and / or motion state of the moving target meets the security hazard alarm requirements.

[0134] In some embodiments, a third processing module is further included, for:

[0135] If at least one image in the image group matches the reference feature, the alarm message generated for the image group is filtered out.

[0136] In some embodiments, a fourth processing module is further included, for:

[0137] The target object is tracked and analyzed based on the image to be processed acquired by at least one first image sensor to obtain the state features of the target object in multiple frames of first images;

[0138] Match the state features of the target object with the security risk alarm requirements in the first images of the multiple frames;

[0139] If the state characteristics of the target object in the first multi-frame image meet the security hazard alarm requirements, the image group is constructed based on the first multi-frame image.

[0140] In some embodiments, a fifth processing module is further included, for:

[0141] Images of the safe area are acquired using multiple secondary image sensors;

[0142] Based on the computer vision model, the target object in the second image acquired by the plurality of second image sensors is subjected to security detection;

[0143] If the safety detection results for the target object in the second images acquired by the plurality of second image sensors indicate that there is a potential production safety hazard and the reliability of the detection results is less than the reliability threshold, the target image is selected from the second images acquired by the plurality of second image sensors.

[0144] In some embodiments, the fifth processing module is specifically used for:

[0145] The image quality is obtained by evaluating the image portion of the target object in the second images acquired by the plurality of second image sensors.

[0146] The second image whose image quality meets the preset requirements is selected as the target image.

[0147] In some embodiments, a sixth processing module is further included, for:

[0148] When the safety detection result of the target object in the second image acquired by the plurality of second image sensors indicates that there is a production safety hazard, and the credibility of the detection result is greater than or equal to the credibility threshold, a safety hazard alarm message for the target object is generated.

[0149] In some embodiments, an adding module is also included for:

[0150] If it is determined that the computer vision model generates a new false detection object, in response to the update request of the whitelist, the reference features extracted by the computer vision model for the new false detection object are obtained;

[0151] Add the reference feature to the whitelist.

[0152] In some embodiments, the first matching module includes:

[0153] The acquisition unit is used to acquire the device identifier of the image sensor that acquires the target image and the pose information of the image sensor;

[0154] The lookup unit is configured to search the cache for a first candidate set contained in the whitelist based on the device identifier and the pose information;

[0155] The first calculation unit is used to determine the similarity between the object features and the reference features of at least one falsely detected object in the first candidate set;

[0156] The matching unit is used to obtain the matching result as a successful match between the object feature and the reference feature when the similarity between the reference feature and the object feature of a falsely detected object in the first candidate set is greater than a preset threshold.

[0157] In some embodiments, a second matching module is further included, for:

[0158] If no reference feature is matched in the first candidate set, the second candidate set contained in the whitelist is searched in the database based on the device identifier and the pose information.

[0159] Determine the similarity between the object features and the object features of at least one falsely detected object in the second candidate set;

[0160] If the similarity between the reference feature of a falsely detected object and the object feature in the second candidate set is greater than a preset threshold, the matching result is that the object feature and the reference feature are successfully matched.

[0161] In some embodiments, an update module is also included, for:

[0162] In response to the update operation of the preset threshold, the preset threshold is updated;

[0163] The initial value of the preset threshold is determined based on the computer vision model.

[0164] In some embodiments, the first matching module includes:

[0165] The construction unit is used to construct an index library corresponding to the reference features in the whitelist;

[0166] The second calculation unit is used to calculate the similarity between the object features and the index library to obtain the matching result.

[0167] In some embodiments, an anomaly detection module is further included, for:

[0168] Anomaly detection is performed on the image state of the image to be processed, and the anomaly detection results are obtained;

[0169] If the anomaly detection result indicates that the image to be processed does not have a preset anomaly, the image to be processed is input into the computer vision model, and the computer vision model performs production safety hazard detection on the target image.

[0170] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0171] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0172] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0173] Figure 10A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0174] like Figure 10 As shown, device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1002 or a computer program loaded from storage unit 1008 into random access memory (RAM) 1003. The RAM 1003 may also store various programs and data required for the operation of device 1000. The computing unit 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004.

[0175] Multiple components in device 1000 are connected to I / O interface 1005, including: input unit 1006, such as keyboard, mouse, etc.; output unit 1007, such as various types of monitors, speakers, etc.; storage unit 1008, such as disk, optical disk, etc.; and communication unit 1009, such as network card, modem, wireless transceiver, etc. Communication unit 1009 allows device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0176] The computing unit 1001 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs the various methods and processes described above, such as image processing methods. For example, in some embodiments, the image processing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, one or more steps of the image processing method described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to perform image processing methods by any other suitable means (e.g., by means of firmware).

[0177] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0178] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0179] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0180] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0181] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0182] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0183] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0184] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An image processing method, comprising: When a computer vision model determines that a target object in a target image poses a production safety hazard, the object features extracted by the computer vision model for the target object are extracted, and the offset between the detection confidence score output by the computer vision model for the target object and the confidence score threshold is less than the offset threshold. The object features are matched with the reference features of falsely detected objects in the whitelist to obtain the matching results; If the object features match the reference features, the detection result of the computer vision model for the target object is corrected to indicate that the target object does not pose a production safety hazard. Determining the target image includes: Images of the safe area are acquired using multiple secondary image sensors; Based on the computer vision model, the target object in the second image acquired by the plurality of second image sensors is subjected to security detection. If the safety detection result of the target object indicates that there is a potential production safety hazard and the credibility of the detection result is less than the credibility threshold, the target image shall be selected from the plurality of second images based at least on image quality.

2. The method according to claim 1, wherein, Determining the target object includes: When the computer vision model determines that multiple candidate objects pose a production safety hazard, the detection confidence level generated by the computer vision model for the multiple candidate objects is obtained. Based on the detection confidence of the multiple candidate objects, candidate objects whose detection confidence is less than the offset threshold are selected as the target objects.

3. The method according to claim 2, further comprising: For the candidate objects among the multiple candidate objects, if the offset between the detection confidence level and the confidence threshold is greater than or equal to the offset threshold, a production safety hazard alarm message is generated for the candidate object.

4. The method according to claim 1, wherein, When the target object is a moving target, the target image is an image in an image group, the image group is the computer vision model performing security detection on the moving target, and the detection result of the security detection is that the motion trajectory and / or motion state of the moving target meets the security hazard alarm requirements.

5. The method according to claim 4, further comprising: If at least one image in the image group matches the reference feature, the alarm message generated for the image group is filtered out.

6. The method according to claim 4, wherein, Determining the image group includes: The target object is tracked and analyzed based on the image to be processed acquired by at least one first image sensor to obtain the state features of the target object in multiple frames of first images; Match the state features of the target object with the security risk alarm requirements in the first images of the multiple frames; If the state characteristics of the target object in the first multi-frame image meet the security hazard alarm requirements, the image group is constructed based on the first multi-frame image.

7. The method of claim 1, further comprising determining the target image based on at least one of the following methods: When the computer vision model determines that the image to be processed poses a production safety hazard, the image to be processed is identified as the target image. If the confidence level is lower than the confidence level threshold, the target image is selected from the second images acquired by the plurality of second image sensors.

8. The method according to claim 1, wherein, The step of filtering the target image from the plurality of second images based at least on image quality includes: The image quality is obtained by evaluating the image portion of the target object in the second images acquired by the plurality of second image sensors. The second image whose image quality meets the preset requirements is selected as the target image.

9. The method according to claim 7, further comprising: When the safety detection result of the target object in the second image acquired by the plurality of second image sensors indicates that there is a production safety hazard, and the credibility of the detection result is greater than or equal to the credibility threshold, a safety hazard alarm message for the target object is generated.

10. The method according to any one of claims 1-9, further comprising: If it is determined that the computer vision model generates a new false detection object, in response to the update request of the whitelist, the reference features extracted by the computer vision model for the new false detection object are obtained; Add the reference feature to the whitelist.

11. The method according to claim 1, wherein, The step of matching the object features with the reference features of falsely detected objects in the whitelist to obtain the matching result includes: Acquire the device identifier of the image sensor that acquires the target image and the pose information of the image sensor; Based on the device identifier and the pose information, search the cache for the first candidate set contained in the whitelist; Determine the similarity between the object features and reference features of at least one falsely detected object in the first candidate set; If the similarity between the reference feature of a falsely detected object and the object feature in the first candidate set is greater than a preset threshold, the matching result is that the object feature and the reference feature are successfully matched.

12. The method of claim 11, further comprising: If no reference feature is matched in the first candidate set, the second candidate set contained in the whitelist is searched in the database based on the device identifier and the pose information. Determine the similarity between the object features and the object features of at least one falsely detected object in the second candidate set; If the similarity between the reference feature of a falsely detected object and the object feature in the second candidate set is greater than a preset threshold, the matching result is that the object feature and the reference feature are successfully matched.

13. The method according to claim 11 or 12, further comprising: In response to the update operation of the preset threshold, the preset threshold is updated; The initial value of the preset threshold is determined based on the computer vision model.

14. The method according to any one of claims 1-9, 11, and 12, wherein, The step of matching the object features with the reference features of falsely detected objects in the whitelist to obtain the matching result includes: Construct an index library corresponding to the reference features in the whitelist; The similarity between the object features and the index is calculated to obtain the matching result.

15. The method according to any one of claims 1-9, 11, and 12, further comprising: Anomaly detection is performed on the image state of the image to be processed, and the anomaly detection results are obtained; If the anomaly detection result indicates that the image to be processed does not have a preset anomaly, the image to be processed is input into the computer vision model, and the computer vision model performs production safety hazard detection on the target image.

16. An image processing apparatus, comprising: The extraction module is used to extract the object features extracted by the computer vision model for the target object when the computer vision model determines that the target object in the target image has a production safety hazard. The offset between the detection confidence score output by the computer vision model of the target object and the confidence score threshold is less than the offset threshold. The first matching module is used to match the object features with the reference features of falsely detected objects in the whitelist to obtain the matching result; The correction module is used to correct the detection result of the computer vision model for the target object to the point that the target object does not pose a production safety hazard when the object features match the reference features. Determining the target image includes: Images of the safe area are acquired using multiple secondary image sensors; Based on the computer vision model, the target object in the second image acquired by the plurality of second image sensors is subjected to security detection. If the safety detection result of the target object indicates that there is a potential production safety hazard and the credibility of the detection result is less than the credibility threshold, the target image shall be selected from the plurality of second images based at least on image quality.

17. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-15.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-15.

19. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-15.

Images