An image foreign matter identification method, device, equipment and storage medium
By recognizing image features through convolutional layers and image segmentation models, the problems of noise-induced false detection and low model scalability in traditional image algorithms are solved, enabling automatic identification and accurate monitoring of any foreign object.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHENZHEN CORERAIN TECH CO LTD
- Filing Date
- 2022-10-21
- Publication Date
- 2026-06-30
AI Technical Summary
Traditional image recognition algorithms suffer from numerous false positives due to noise, and pre-trained models are unable to effectively identify newly added foreign objects, exhibiting low scalability.
Image features are extracted by convolutional layers and the segmentation results of the feature maps are identified using a pre-trained image segmentation model. The differences in the segmentation results are compared to determine foreign object information without needing to focus on the type of foreign object.
It effectively reduces environmental interference, improves identification accuracy and efficiency, and can automatically identify any type of foreign object, making it suitable for accurate monitoring in scenarios such as fire exits.
Smart Images
Figure CN115661475B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image recognition technology, and in particular to a method, apparatus, device, and storage medium for identifying foreign objects in images. Background Technology
[0002] Traditional image recognition algorithms obtain foreign object features by comparing the differences between two images, and then identify the obstructing foreign object through a series of filtering and weighted summations. However, manually extracted features are relatively limited, and even with various filtering methods, noise cannot be filtered out, resulting in a large number of false detections in the final image. These false detections include issues such as changes in lighting and shadows, ground textures, and fluctuations in camera pixel noise.
[0003] Deep learning-based image algorithms often employ object detection and semantic segmentation for image recognition. This method effectively addresses the problem of high false detection rates in traditional algorithms, but it is only effective for specific objects, such as garbage, cardboard boxes from express delivery, and cars. This requires prior knowledge of the object type and training the model based on that type before using the trained model for recognition. It struggles to recognize new types of foreign objects and has low scalability. Summary of the Invention
[0004] This invention provides an image foreign object recognition method, apparatus, device, and storage medium to solve the problems of high false detection rates caused by noise still present in manually extracted features, and the inability of pre-trained models to effectively identify newly added foreign objects.
[0005] In a first aspect, the present invention provides an image foreign object identification method, the method comprising:
[0006] The process involves acquiring an image to be inspected and a preset template image, extracting a first feature map of the image to be inspected through a preset convolutional layer, and extracting a second feature map of the preset template image through the preset convolutional layer.
[0007] A first segmentation result of the first feature map is identified by a pre-trained image segmentation model, and a second segmentation result of the second feature map is identified by the image segmentation model.
[0008] Compare the first segmentation result and the second segmentation result; if the first segmentation result and the second segmentation result are different, obtain the first difference information between the first segmentation result and the second segmentation result; determine the first foreign object information of the image to be inspected based on the first difference information;
[0009] If the first segmentation result is the same as the second segmentation result, then it is determined that there are no foreign objects.
[0010] In a second aspect, the present invention provides a control device including a unit for performing an image foreign object identification method as described in any embodiment of the first aspect.
[0011] Thirdly, an electronic device is provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;
[0012] Memory, used to store computer programs;
[0013] When a processor executes a program stored in a memory, it implements the steps of the image foreign object recognition method according to any embodiment of the first aspect.
[0014] Fourthly, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the image foreign object recognition method as described in any embodiment of the first aspect.
[0015] The technical solutions provided in the embodiments of the present invention have the following advantages compared with the prior art:
[0016] The method provided in this invention extracts features from a template image and the image to be inspected using a convolutional layer to obtain a feature map. This effectively reduces environmental interference and enables automatic feature extraction, significantly improving accuracy and efficiency compared to manual feature extraction. The method uses an image segmentation model to identify the first and second segmentation results of the image to be inspected, and determines the first foreign object information based on the first difference information between the first and second segmentation results. This invention's method does not require attention to the type of foreign object; it only focuses on the presence of a foreign object. If the first and second segmentation results are different, a foreign object is determined to be present in the image to be inspected. Therefore, regardless of the type of foreign object, the technical solution of this invention can identify its presence. In contrast, existing methods require specific classification of foreign objects into a particular category, thus failing to identify new foreign object categories.
[0017] The technical solution of this invention is particularly applicable to the scenario of foreign object identification in fire lanes. In the case of fire lanes, monitoring personnel usually only pay attention to whether there are foreign objects in the fire lanes, and do not pay much attention to the type of foreign objects. Through the technical solution of this invention, it is possible to accurately identify whether there are foreign objects in the fire lanes. When foreign objects are present, an alarm message can be issued to the monitoring personnel, thereby achieving timely and accurate monitoring. Attached Figure Description
[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.
[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0020] Figure 1 This is a flowchart illustrating an image foreign object recognition method provided in Embodiment 1 of the present invention;
[0021] Figure 2 This is a flowchart illustrating an image foreign object recognition method provided in Embodiment 2 of the present invention;
[0022] Figure 3 This is a schematic diagram of a sub-process of an image foreign object recognition method provided in an embodiment of the present invention;
[0023] Figure 4 This is a schematic diagram of a sub-process of an image foreign object recognition method provided in an embodiment of the present invention;
[0024] Figure 5 This is a schematic diagram of a control device structure provided in Embodiment 1 of the present invention;
[0025] Figure 6 This is a schematic diagram of a control device structure provided in Embodiment 2 of the present invention;
[0026] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention;
[0027] Figure 8 Example diagrams provided for embodiments of the present invention. Detailed Implementation
[0028] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0029] Example 1
[0030] Figure 1 This is a flowchart illustrating an image foreign object identification method provided by an embodiment of the present invention. The present invention proposes an image foreign object identification method; specifically, see [link to relevant documentation]. Figure 1 The image foreign object identification method includes the following steps S101-S104.
[0031] S101, acquire the image to be inspected, extract the first feature map of the image to be inspected through a preset convolutional layer, and extract the second feature map of the preset template image through the preset convolutional layer.
[0032] In specific implementation, frame images can be extracted from the video stream captured by the video acquisition device as the image to be inspected, or the image to be inspected can be input by the user. It should be noted that the above method of obtaining the image to be inspected is merely an example; those skilled in the art can also use other methods to obtain the image to be inspected, which will not exceed the protection scope of this invention. The image to be inspected is convolved through a preset convolutional layer, and the resulting feature map is used as the first feature map; the template image is convolved through a preset convolutional layer, and the resulting feature map is used as the second feature map. The specific principle of the convolutional calculation can be determined with reference to existing materials, and this invention does not specifically limit it.
[0033] By performing convolution calculations on the image to be inspected and the template image using convolutional layers, the influence of uncertain factors such as noise and lighting can be avoided, and the object contours in the image to be inspected and the template image can be extracted more clearly, so that the feature map can be segmented by the image segmentation model in the subsequent process.
[0034] In one embodiment, step S101 specifically includes: extracting frame images from the video stream of a preset video acquisition device as the image to be inspected.
[0035] In practice, the video acquisition device includes a camera. A camera typically has basic functions such as video recording / transmission and still image capture. It captures images through a lens, and then the image is processed by the camera's internal photosensitive components and control components, converting it into a digital signal that a computer can recognize. This signal is then input to a computer via a parallel port or USB connection, where software performs image reconstruction. Frames from the video stream are extracted as the images to be inspected.
[0036] S102, identify the first segmentation result of the first feature map through a pre-trained image segmentation model, and identify the second segmentation result of the second feature map through the image segmentation model.
[0037] In practical implementation, image segmentation models divide a complete image into several parts according to a set method or rule, and assign a "label" to each part according to a certain rule. However, the disadvantage is that the number of identifiable objects is small, and when the number of identifiable objects needs to be increased, a large amount of data needs to be collected and the model retrained. In this invention, only the function of image segmentation models to divide a complete object into several parts according to a certain method or rule is utilized. By recognizing the first feature map through the image segmentation model, all object contours in the first feature map can be distinguished and segmented. Thus, the first segmentation result includes the coordinate information, pixel information, and area information of all object contours. Further, by recognizing the second feature map through the image segmentation model, all object contours in the second feature map can be distinguished and segmented. Thus, the second segmentation result includes the coordinate information, pixel information, and area information of all object contours. It should be noted that the object differentiation is not based on the type of object, but on the obtained object contours and the image area bounded by the object contours in the image to be inspected.
[0038] By recognizing, distinguishing, and segmenting object contours through an image segmentation model, new foreign objects can be quickly and accurately identified without prior training on the recognition capabilities based on the types of foreign objects collected. This results in a high iteration frequency.
[0039] S103, compare the first segmentation result and the second segmentation result; if the first segmentation result and the second segmentation result are different, obtain the first difference information between the first segmentation result and the second segmentation result; determine the first foreign object information of the image to be inspected based on the first difference information.
[0040] In specific implementation, the first segmentation result is compared with the second segmentation result to obtain the first difference information. The first difference information is specifically the object contour unique to the first segmentation result. For example, in one embodiment, after comparing the first segmentation result with the second segmentation result, it is found that the first object contour and the second object contour exist in the first segmentation result but not in the second segmentation result. In this case, the first object contour and the second object contour are used as the first difference information.
[0041] The first difference information obtained by comparing the first segmentation result with the second segmentation result is used to determine the first foreign object information in the image to be inspected. For example, in one embodiment, the first difference information includes a first object contour, and the image area bounded by the first object contour in the image to be inspected is determined as the first foreign object information. The first foreign object information corresponds to the object contour included in the first difference information. If the first difference information includes multiple object contours, multiple first foreign object information can be determined one-to-one within the image area based on the multiple object contours.
[0042] In one embodiment, after step S103 above, the method further includes: marking the first foreign object information in the image to be inspected.
[0043] In specific implementation, the first foreign object information corresponds to the outline of the first object. The image area defined by the outline of the first object in the image to be inspected can be marked in the image to be inspected. The marking can be in the form of a square, a circle, etc., and the marking color can be red, green, blue, etc.
[0044] See Figure 8 The image detects multiple object outlines and marks the defined image area in the image to be inspected with a box shape. The number of pixels of the foreign object corresponding to the defined image is marked above the box shape.
[0045] S104, if the first segmentation result is the same as the second segmentation result, then it is determined that there are no foreign objects.
[0046] In practice, if the first segmentation result is compared with the second segmentation result and no first difference information is obtained, then no first foreign object information is obtained, and it is determined that the image to be inspected does not contain any foreign objects compared with the template image.
[0047] Feature maps are obtained by extracting features from the template image and the image to be inspected through convolutional layers. This effectively reduces environmental interference and enables automatic feature extraction, significantly improving accuracy and efficiency compared to manual feature extraction. The image segmentation model identifies the first and second segmentation results of the image to be inspected, and determines the first foreign object information based on the first difference information between the two results. This invention does not require attention to the type of foreign object, only to its presence. If the first and second segmentation results are different, a foreign object is determined to be present in the image to be inspected. Therefore, regardless of the type of foreign object, this invention can identify its presence. In contrast, existing methods require specific classification of foreign objects into a particular category, thus failing to identify new categories of foreign objects. This invention is particularly suitable for foreign object identification in fire escape routes. In fire escape routes, monitoring personnel typically only focus on the presence of foreign objects, without paying much attention to their type. This invention accurately identifies the presence of foreign objects in fire escape routes and issues alarms to monitoring personnel when they are present, enabling timely and accurate monitoring.
[0048] Example 2
[0049] Figure 2 This is a flowchart illustrating an image foreign object identification method provided by an embodiment of the present invention. The present invention proposes an image foreign object identification method; specifically, see [link to relevant documentation]. Figure 2 The image foreign object identification method includes the following steps S201-S215.
[0050] S201, extract frame images from the video stream of the preset video acquisition device as the initial template image.
[0051] In practice, the video acquisition device operates normally and can capture video within the range monitored by the camera. After the staff clears the foreign objects within the range monitored by the camera, the device begins to extract frame images from the video stream as initial template images.
[0052] By having staff pre-emptively remove foreign objects from the monitored area, it is ensured that no foreign objects are present in the extracted frame images from the video stream. This allows for the acquisition of more authoritative images as reference images for identifying foreign objects, thus obtaining an accurate template image free of foreign objects.
[0053] S202, extract the next frame image of the initial template image from the video stream as a comparison image.
[0054] In practice, the next frame of the initial template image is extracted from the video stream as a comparison image.
[0055] S203, determine whether the comparison image is the same as the initial template image.
[0056] In practice, the comparison image is compared with the initial template image. The presence of unstable factors is determined based on whether the comparison image is identical to the initial template image. If the comparison image is identical to the initial template image, it is determined that no unstable factors are present; if the comparison image is different from the initial template image, it is determined that unstable factors are present. By determining whether the comparison image is identical to the initial template image, it is possible to effectively avoid using images acquired in unstable states as template images, thus preventing the acquisition of template images containing foreign objects and affecting subsequent foreign object identification.
[0057] In one embodiment, see Figure 4 , Figure 4 This is a schematic diagram of a sub-process of an image foreign object recognition method provided in an embodiment of the present invention. Step S203 above includes steps S401-S404:
[0058] S401, extract the fourth feature map of the initial template map through the preset convolutional layer, and extract the fifth feature map of the comparison map through the preset convolutional layer.
[0059] In specific implementation, the initial template image is convolved through a preset convolutional layer to obtain a feature map, which is used as the fourth feature map; the comparison image is convolved through a preset convolutional layer to obtain a feature map, which is used as the fifth feature map.
[0060] S402, the fourth segmentation result of the fourth feature map is identified by the image segmentation model, and the fifth segmentation result of the fifth feature map is identified by the image segmentation model.
[0061] In specific implementation, the image segmentation model has the function of recognizing objects in feature images and depicting object contours. By recognizing the fourth feature image using the image segmentation model, all object contours in the fourth feature image can be distinguished and segmented. Therefore, the fourth segmentation result includes the coordinate information, pixel information, and area information of all object contours. Similarly, by recognizing the fifth feature image using the image segmentation model, all object contours in the fifth feature image can be distinguished and segmented. Therefore, the fifth segmentation result includes the coordinate information, pixel information, and area information of all object contours. It should be noted that the object differentiation is not based on object type, but rather on the obtained object contours and the image regions defined by the object contours in the image to be inspected.
[0062] S403, if the fourth segmentation result is different from the fifth segmentation result, it is determined that the comparison image is different from the initial template image.
[0063] In practice, when the fourth segmentation result is different from the fifth segmentation result, it is usually due to unstable factors such as moving objects in the camera area, camera flickering noise, and lens shaking. Therefore, the image obtained in the current unstable state cannot be used as a template image correctly. It should be determined that the comparison image is different from the initial template image to avoid using the image obtained in the current unstable state as a template image.
[0064] S404, if the fourth segmentation result is the same as the fifth segmentation result, it is determined that the comparison image is the same as the initial template image.
[0065] In specific implementation, when the fourth segmentation result is the same as the fifth segmentation result, it is determined that no unstable factors have been detected, that is, the acquired frame image is stable, and then it is determined that the comparison image is the same as the initial template image.
[0066] S204, if the comparison image is the same as the initial template image, obtain the number of times the target step is executed, and determine whether the number of times the target step is executed is less than a preset number threshold. The target step is the step of determining whether the comparison image is the same as the initial template image.
[0067] In specific implementation, if the comparison image is the same as the initial template image, the number of times the target step S203 is executed is recorded. Recording the number of times the comparison image is acquired ensures that multiple comparisons are performed between the comparison image and the initial template image. This avoids the influence of unstable factors such as moving objects, camera flickering noise, and lens shaking on the initial template image, rigorously ensuring that the template image is obtained from a stable state.
[0068] S205, if the number of times the target step is executed is less than a preset threshold, the comparison image is used as a new initial template image, and the process jumps to step S202.
[0069] For example, in one embodiment, it is required that no foreign object is detected in 10 consecutive comparison images, that is, the preset number threshold is 10. If the number of times the target step S203 is executed is less than 10, the previous comparison image in which no foreign object was detected is used as the new initial template image, and the process jumps to step S202.
[0070] S206, if the number of times the target step is executed is equal to the number threshold, the comparison image is used as the template image.
[0071] For example, in one embodiment, it is required that no foreign object is detected in 10 consecutive comparison images, that is, the preset number threshold is 10. If the number of times the target step S203 is executed is equal to 10, then the 10th comparison image is used as the template image.
[0072] If the comparison image is different from the initial template image, proceed to step S201.
[0073] In specific implementation, if the comparison image is different from the initial template image, it is determined that the comparison image has detected a foreign object, and the comparison image cannot be used as a template image. In one embodiment, it is required that no foreign object is detected in 10 consecutive comparison images. If the comparison image is different from the initial template image before the 10th execution of the target step S203, it is determined that the current comparison image has detected a foreign object, and it is necessary to jump to step S201 and re-record the number of times the target step S203 has been executed.
[0074] Using a comparison image of a series of frames in which no foreign object was detected as a template image makes it easier to accurately identify foreign objects in the future.
[0075] S207, acquire the image to be inspected and a preset template image, extract the first feature map of the image to be inspected through a preset convolutional layer, and extract the second feature map of the preset template image through the preset convolutional layer.
[0076] In specific implementation, frame images can be extracted from the video stream captured by the video acquisition device as the image to be inspected, or the image to be inspected can be input by the user. It should be noted that the above method of obtaining the image to be inspected is merely an example; those skilled in the art can also use other methods to obtain the image to be inspected, which will not exceed the protection scope of this invention. The image to be inspected is convolved through a preset convolutional layer, and the resulting feature map is used as the first feature map; the template image is convolved through a preset convolutional layer, and the resulting feature map is used as the second feature map. The specific principle of the convolutional calculation can be determined with reference to existing materials, and this invention does not specifically limit it.
[0077] By performing convolution calculations on the image to be inspected and the template image using convolutional layers, the influence of uncertain factors such as noise and lighting can be avoided, and the object contours in the image to be inspected and the template image can be extracted more clearly, so that the feature map can be segmented by the image segmentation model in the subsequent process.
[0078] In one embodiment, step S207 specifically includes: extracting frame images from the video stream of a preset video acquisition device as the image to be inspected.
[0079] In practice, the video acquisition device includes a camera. A camera typically has basic functions such as video recording / transmission and still image capture. It captures images through a lens, and then the image is processed by the camera's internal photosensitive components and control components, converting it into a digital signal that a computer can recognize. This signal is then input to a computer via a parallel port or USB connection, where software performs image reconstruction. Frames from the video stream are extracted as the images to be inspected.
[0080] S208, identify the first segmentation result of the first feature map through a pre-trained image segmentation model, and identify the second segmentation result of the second feature map through the image segmentation model.
[0081] In practical implementation, image segmentation models divide a complete image into several parts according to a set method or rule, and assign a "label" to each part according to a certain rule. However, the disadvantage is that the number of identifiable objects is small, and when the number of identifiable objects needs to be increased, a large amount of data needs to be collected and the model retrained. In this invention, only the function of image segmentation models to divide a complete object into several parts according to a certain method or rule is utilized. By recognizing the first feature map through the image segmentation model, all object contours in the first feature map can be distinguished and segmented. Thus, the first segmentation result includes the coordinate information, pixel information, and area information of all object contours. Further, by recognizing the second feature map through the image segmentation model, all object contours in the second feature map can be distinguished and segmented. Thus, the second segmentation result includes the coordinate information, pixel information, and area information of all object contours. It should be noted that the object differentiation is not based on the type of object, but on the obtained object contours and the image area bounded by the object contours in the image to be inspected.
[0082] By recognizing, distinguishing, and segmenting object contours through an image segmentation model, new foreign objects can be quickly and accurately identified without prior training on the recognition capabilities based on the types of foreign objects collected. This results in a high iteration frequency.
[0083] S209, compare the first segmentation result and the second segmentation result; if the first segmentation result and the second segmentation result are different, obtain the first difference information between the first segmentation result and the second segmentation result; determine the first foreign object information of the image to be inspected based on the first difference information.
[0084] In specific implementation, the first segmentation result is compared with the second segmentation result to obtain the first difference information. The first difference information is specifically the object contour unique to the first segmentation result. For example, in one embodiment, after comparing the first segmentation result with the second segmentation result, it is found that the first object contour and the second object contour exist in the first segmentation result but not in the second segmentation result. In this case, the first object contour and the second object contour are used as the first difference information.
[0085] The first difference information obtained by comparing the first segmentation result with the second segmentation result is used to determine the first foreign object information in the image to be inspected. For example, in one embodiment, the first difference information includes a first object contour, and the image area bounded by the first object contour in the image to be inspected is determined as the first foreign object information. The first foreign object information corresponds to the object contour included in the first difference information. If the first difference information includes multiple object contours, multiple first foreign object information can be determined one-to-one within the image area based on the multiple object contours.
[0086] In one embodiment, after step S209 above, the method further includes: marking the first foreign object information in the image to be inspected.
[0087] In specific implementation, the first foreign object information corresponds to the outline of the first object. The image area defined by the outline of the first object in the image to be inspected can be marked in the image to be inspected. The marking can be in the form of a square, a circle, etc., and the marking color can be red, green, blue, etc.
[0088] See Figure 8 The image detects multiple object outlines and marks the defined image area in the image to be inspected with a box shape. The number of pixels of the foreign object corresponding to the defined image is marked above the box shape.
[0089] S210, if the first segmentation result is the same as the second segmentation result, then it is determined that there are no foreign objects.
[0090] In practice, if the first segmentation result is compared with the second segmentation result and no first difference information is obtained, then no first foreign object information is obtained, and it is determined that the image to be inspected does not contain any foreign objects compared with the template image.
[0091] S211, extract a preset number of target images from the video stream following the image to be inspected as a verification image set.
[0092] For example, in one embodiment, six target images following the image to be inspected are extracted from the video stream as a verification image set.
[0093] S212, Obtain the second foreign object information of each target image in the verification image set.
[0094] For example, in one embodiment, the six target images in the verification image set are convolved and then input together with the feature images of the template image after convolving and input into the image segmentation model. The image segmentation model outputs the second foreign object information of the six target images in the verification image set.
[0095] In one embodiment, see Figure 3 , Figure 3This is a schematic diagram of a sub-process of an image foreign object recognition method provided in an embodiment of the present invention. Step S212 above includes steps S301-S304:
[0096] In specific implementation, the following description is based on one of the six target images in the verification image set. In the actual scenario, all target images in the verification image set will undergo the following steps S301-S304.
[0097] S301, the third feature map of the target image is extracted through the preset convolutional layer.
[0098] In a specific implementation, the target image is convolved through a preset convolutional layer to calculate the feature map obtained, which is then used as the third feature map.
[0099] S302, the third segmentation result of the third feature map is identified through the image segmentation model.
[0100] In practice, the third feature map is identified using an image segmentation model, which distinguishes all object contours in the third feature map and segments them. Therefore, the third segmentation result includes the coordinate information, pixel information, and area information of all object contours.
[0101] S303, if the third segmentation result is different from the second segmentation result, obtain the second difference information between the third segmentation result and the second segmentation result.
[0102] In specific implementation, the third segmentation result is compared with the second segmentation result to obtain the second difference information. The second difference information is specifically the object contour unique to the third segmentation result. For example, in one embodiment, after comparing the third segmentation result with the second segmentation result, it is found that the third object contour and the fourth object contour exist in the third segmentation result but not in the second segmentation result. In this case, the third object contour and the fourth object contour are used as the second difference information.
[0103] S304, determine the second foreign object information of the target image based on the second difference information.
[0104] In specific implementation, the second difference information obtained by comparing the third segmentation result with the second segmentation result is used to determine the second foreign object information in the image to be inspected. For example, in one embodiment, the second difference information includes a third object contour, then the image area bounded by the third object contour in the image to be inspected is determined as the second foreign object information. The second foreign object information corresponds to the object contour included in the second difference information. If the second difference information includes multiple object contours, then multiple pieces of second foreign object information can be determined one-to-one within the image area based on the multiple object contours.
[0105] S213, obtain the proportion of target images in the verification image set that have the same second foreign object information as the first foreign object information.
[0106] In specific implementation, after performing steps S301-S304 on all target images in the verification image set, the proportion of target images in the verification image set whose second foreign object information is the same as the first foreign object information is calculated. For example, in one embodiment, the verification image set includes 6 target images, and 4 of them contain target images whose second foreign object information is the same as the first foreign object information. Therefore, the proportion of target images in the verification image set whose second foreign object information is the same as the first foreign object information is 66.7%.
[0107] S214, if the ratio is less than a preset ratio threshold, the first foreign object information is determined to be a misjudgment.
[0108] For example, in one embodiment, the preset ratio threshold is 66.6%. In the above 6 target images, if 4 frames contain the same second foreign object information as the first foreign object information, representing a ratio of 66.7%, the first foreign object information is determined to be normal, i.e., the first foreign object information is identified as a foreign object. If 3 frames contain the same second foreign object information as the first foreign object information, representing a ratio of 50%, and this ratio is less than the preset ratio threshold of 66.7%, the first foreign object information is determined to be a misjudgment, i.e., the first foreign object information is not identified as a foreign object, and the misjudged first foreign object information will be filtered out. This ratio further verifies the identification of foreign objects, avoiding misjudgments caused by moving objects and camera flicker noise.
[0109] Example 3
[0110] See Figure 5 This invention also provides an image foreign object recognition device 500, which includes a first acquisition unit 501, a first recognition unit 502, a first comparison unit 503, and a first determination unit 504.
[0111] The first acquisition unit 501 is used to acquire an image to be inspected and a preset template image, extract a first feature map of the image to be inspected through a preset convolutional layer, and extract a second feature map of the preset template image through the preset convolutional layer.
[0112] In one embodiment, the first acquisition unit 501 specifically includes:
[0113] Frame images are extracted from the video stream of a preset video acquisition device as the image to be inspected.
[0114] The first recognition unit 502 is used to recognize the first segmentation result of the first feature map through a pre-trained image segmentation model, and to recognize the second segmentation result of the second feature map through the image segmentation model.
[0115] The first comparison unit 503 is used to compare the first segmentation result and the second segmentation result; if the first segmentation result and the second segmentation result are different, the first difference information between the first segmentation result and the second segmentation result is obtained; and the first foreign object information of the image to be inspected is determined based on the first difference information.
[0116] The first determination unit 504 is used to determine that there are no foreign objects if the first segmentation result is the same as the second segmentation result.
[0117] Example 4
[0118] See Figure 6 This invention also provides an image foreign object recognition device 600, which includes a first extraction unit 601, a second extraction unit 602, a first judgment unit 603, a second acquisition unit 604, a third jump unit 605, a second judgment unit 606, a third acquisition unit 607, a second recognition unit 608, a second comparison unit 609, a third judgment unit 610, a third extraction unit 611, a fourth acquisition unit 612, a fifth acquisition unit 613, and a fourth judgment unit 614.
[0119] The first extraction unit 601 is used to extract frame images from the video stream of a preset video acquisition device as initial template images.
[0120] The second extraction unit 602 is used to extract the next frame image of the initial template image from the video stream as a comparison image.
[0121] The first judgment unit 603 is used to determine whether the comparison image is the same as the initial template image.
[0122] The second acquisition unit 604 is used to acquire the number of times the target step is executed if the comparison image is the same as the initial template image, and to determine whether the number of times the target step is executed is less than a preset number threshold. The target step is the step of determining whether the comparison image is the same as the initial template image.
[0123] The third jump unit 605 is used to, if the number of times the target step is executed is less than a preset number threshold, use the comparison image as a new initial template image and jump to the second extraction unit 602.
[0124] The second determination unit 606 is used to use the comparison image as the template image if the number of times the target step is executed is equal to the number threshold.
[0125] In one embodiment, if the comparison image is different from the initial template image, the process jumps to the first extraction unit 601.
[0126] The third acquisition unit 607 is used to acquire the image to be inspected and a preset template image, extract a first feature map of the image to be inspected through a preset convolutional layer, and extract a second feature map of the preset template image through the preset convolutional layer.
[0127] In one embodiment, the third acquisition unit 607 specifically includes:
[0128] Frame images are extracted from the video stream of a preset video acquisition device as the image to be inspected.
[0129] The second recognition unit 608 is used to recognize the first segmentation result of the first feature map through a pre-trained image segmentation model, and to recognize the second segmentation result of the second feature map through the image segmentation model.
[0130] The second comparison unit 609 is used to compare the first segmentation result and the second segmentation result; if the first segmentation result and the second segmentation result are different, the first difference information between the first segmentation result and the second segmentation result is obtained; and the first foreign object information of the image to be inspected is determined based on the first difference information.
[0131] The third determination unit 610 is used to determine that there are no foreign objects if the first segmentation result is the same as the second segmentation result.
[0132] The third extraction unit 611 is used to extract a preset number of target images from the video stream following the image to be inspected as a verification image set.
[0133] The fourth acquisition unit 612 is used to acquire the second foreign object information of each target image in the verification image set.
[0134] In one embodiment, the fourth acquisition unit 612 specifically includes:
[0135] The third feature map of the target image is extracted through the preset convolutional layer;
[0136] The third segmentation result of the third feature map is identified by the image segmentation model;
[0137] If the third segmentation result is different from the second segmentation result, obtain the second difference information between the third segmentation result and the second segmentation result;
[0138] The second foreign object information of the target image is determined based on the second difference information.
[0139] The fifth acquisition unit 613 is used to acquire the proportion of target images in the verification image set that have the same second foreign object information as the first foreign object information.
[0140] The fourth determination unit 614 is used to determine that the first foreign object information is a misjudgment if the ratio is less than a preset ratio threshold.
[0141] like Figure 7 As shown, this embodiment of the invention provides an electronic device, including a processor 111, a communication interface 112, a memory 113, and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.
[0142] Memory 113 is used to store computer programs;
[0143] In one embodiment of the present invention, the processor 111, when executing the program stored in the memory 113, implements the control method of the image foreign object recognition method provided in any of the foregoing method embodiments.
[0144] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the image foreign object identification method provided in any of the foregoing method embodiments.
[0145] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0146] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.
Claims
1. An image foreign matter recognition method characterized by comprising: include: The process involves acquiring an image to be inspected and a preset template image, extracting a first feature map of the image to be inspected through a preset convolutional layer, and extracting a second feature map of the preset template image through the preset convolutional layer. A first segmentation result of the first feature map is identified by a pre-trained image segmentation model, and a second segmentation result of the second feature map is identified by the image segmentation model. Compare the first segmentation result with the second segmentation result; If the first segmentation result is different from the second segmentation result, first difference information between the first segmentation result and the second segmentation result is obtained; first foreign object information of the image to be inspected is determined based on the first difference information. If the first segmentation result is the same as the second segmentation result, then it is determined that there are no foreign objects. After determining the foreign object information of the image to be inspected based on the first difference information, the method further includes: A preset number of target images are extracted from the image to be inspected from the preset video stream as a verification image set; Obtain the second foreign object information of each target image in the verification image set; Obtain the proportion of target images in the verification image set whose second foreign object information is the same as the first foreign object information; If the ratio is less than a preset ratio threshold, the first foreign object information is determined to be a false judgment. The method further includes, prior to acquiring the image to be inspected, extracting a first feature map of the image to be inspected through a preset convolutional layer, and extracting a second feature map of a preset template image through the preset convolutional layer: Extract frame images from the video stream of the preset video acquisition device as the initial template image; Extract the next frame of the initial template image from the video stream as a comparison image; Determine whether the comparison image is the same as the initial template image; If the comparison image is the same as the initial template image, the number of times the target step is executed is obtained, and it is determined whether the number of times the target step is executed is less than a preset threshold. The target step is the step of determining whether the comparison image is the same as the initial template image. If the number of times the target step is executed is less than a preset threshold, the comparison image is used as a new initial template image, and the process jumps to the step of extracting the next frame image of the initial template image from the video stream as the comparison image. If the number of times the target step is executed is equal to the number threshold, the comparison image is used as the template image; If the comparison image is different from the initial template image, proceed to the step of extracting frame images from the video stream of the preset video acquisition device as the initial template image.
2. The image foreign matter recognition method according to claim 1, characterized by, The acquisition of the image to be inspected includes: Frame images are extracted from the video stream of a preset video acquisition device as the image to be inspected.
3. The image foreign matter recognition method according to claim 1, characterized by, The step of obtaining the second foreign object information for each target image in the verification image set includes: The third feature map of the target image is extracted through the preset convolutional layer; The third segmentation result of the third feature map is identified by the image segmentation model; If the third segmentation result is different from the second segmentation result, obtain the second difference information between the third segmentation result and the second segmentation result; The second foreign object information of the target image is determined based on the second difference information.
4. The image foreign matter recognition method according to claim 1, characterized by, The step of determining whether the comparison image is the same as the initial template image includes: The fourth feature map of the initial template map is extracted through the preset convolutional layer, and the fifth feature map of the comparison map is extracted through the preset convolutional layer; The image segmentation model is used to identify the fourth segmentation result of the fourth feature map and the image segmentation model is used to identify the fifth segmentation result of the fifth feature map. If the fourth segmentation result is different from the fifth segmentation result, it is determined that the comparison image is different from the initial template image. If the fourth segmentation result is the same as the fifth segmentation result, it is determined that the comparison image is the same as the initial template image.
5. The method of claim 1, wherein, After determining the first foreign object information of the image to be inspected based on the first difference information, the method further includes: The information of the first foreign object is marked in the image to be inspected.
6. An image foreign matter recognition apparatus characterized by comprising: Includes a unit for performing the method as described in any one of claims 1-5.
7. An electronic device, comprising: It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the method according to any one of claims 1-5.
8. A computer-readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-5.