Foreign matter detection method, foreign matter detection device, equipment and storage medium

Abstract

The present application discloses a foreign body detection method, foreign body detection device, equipment and storage medium, the method comprising: obtaining images to be detected corresponding to a plurality of target image frames in a target video, the target video being acquired from a commodity placement area; using a foreign body detection model, performing foreign body detection on each image to be detected, obtaining a foreign body judgment score for each image to be detected, the foreign body judgment score representing the probability that the image to be detected contains a foreign body, and the foreign body is an object other than a commodity; determining the foreign body detection result of the target video based at least on the foreign body judgment score of each image to be detected, the foreign body detection result being whether or not a foreign body is contained. The above scheme can realize foreign body detection in commodity placement areas.

Description

Technical Field

[0001] The present application relates to the field of computer vision processing, and in particular to a foreign body detection method, a foreign body detection device, a device and a storage medium. Background Art

[0002] The application of devices for placing goods, such as vending machines and automatic selling machines, has gradually become widespread. These devices have one or more goods placed inside and can be used for selling goods, etc. Users can take objects from or put objects into the devices by themselves.

[0003] In some cases, users may put foreign objects that are different from the goods stored in the device. The presence of foreign objects will bring certain risks to the operation of the device, for example, they may be mistaken for goods by other users, disrupting the sales order, etc. However, it is not possible to detect whether there are foreign objects in the device in time. Therefore, how to detect foreign objects in the device storing goods has become an urgent problem to be solved. Summary of the invention

[0004] The present application at least provides a foreign matter detection method, a foreign matter detection device, a device and a storage medium.

[0005] The present application provides a foreign body detection method, comprising: obtaining images to be detected corresponding to a plurality of target image frames in a target video, wherein the target video is acquired from a commodity placement area; using a foreign body detection model, performing foreign body detection on each image to be detected, and obtaining a foreign body judgment score of each image to be detected, wherein the foreign body judgment score represents the probability that the image to be detected contains a foreign body, and the foreign body is an object other than a commodity; and determining a foreign body detection result of the target video based at least on the foreign body judgment score of each image to be detected, wherein the foreign body detection result is whether the foreign body is contained or not contained.

[0006] Therefore, a target video of the commodity placement area is obtained, and a foreign object detection model is used to detect foreign objects in the image to be detected contained in the target video to obtain the probability that the image contains foreign objects, thereby determining whether the target video contains foreign objects.

[0007] Among them, determining the foreign matter detection result of the target video at least based on the foreign matter judgment score of each image to be detected includes: obtaining several types of statistical factors, wherein at least one type of statistical factor is determined based on the foreign matter judgment score of the image to be detected; based on several types of statistical factors, determining the foreign matter detection result of the target video, wherein, when any type of statistical factor meets the corresponding foreign matter detection condition, the foreign matter detection result of the target video is that it does not contain foreign matter, or when all types of statistical factors meet the corresponding foreign matter detection condition, the foreign matter detection result of the target video is that it does not contain foreign matter.

[0008] Therefore, at least one type of statistical factor is determined based on the foreign matter judgment score, which can reflect the presence of foreign matter in the target video. Therefore, by using several types of statistical factors, it is possible to determine whether the target video contains foreign matter.

[0009] Among them, several types of statistical factors include at least one of a first type of statistical factors and a second type of statistical factors; wherein the first type of statistical factors include at least one of a first statistical value of the number of target image frames and a commodity classification score of all images to be detected, and a second statistical value of the foreign matter judgment score of all images to be detected, and the commodity classification score represents the probability that the image to be detected contains commodities of a preset commodity category.

[0010] Therefore, the first statistical value of the commodity classification score and the number of target image frames are both statistical factors about the target video, which can be combined with the second statistical value of the foreign object judgment score to reflect the characteristics of the target video. They can be used to determine the foreign object situation in the target video, and then determine whether the target video contains foreign objects.

[0011] Among them, the second type of statistical factors is the target ratio of the number of images to be detected that are suspected of containing foreign objects to the total number of target image frames. The images to be detected that are suspected of containing foreign objects are determined by comparing the foreign object judgment score of the images to be detected with the score threshold.

[0012] Therefore, the target ratio of the number of images to be detected that are suspected of containing foreign objects to the total number of target image frames can reflect the foreign object situation of the target video, and then determine whether the target video contains foreign objects.

[0013] Among them, in response to the fact that several types of statistical factors include the second type of statistical factors, and the second type of statistical factors are target ratios, by judging whether the target ratio is within a preset ratio range, it is determined whether the second type of statistical factors meet the corresponding foreign object detection conditions; in response to the fact that several types of statistical factors include the first type of statistical factors, and the first type of statistical factors include the second statistical value, by judging whether the second statistical value is within a first preset numerical range, it is determined whether the first type of statistical factors meet the corresponding foreign object detection conditions; in response to the fact that several types of statistical factors include the first type of statistical factors, and the first type of statistical factors include the first statistical value, by judging whether the first statistical value is within a second preset numerical range, it is determined whether the first type of statistical factors meet the corresponding foreign object detection conditions; in response to the fact that several types of statistical factors include the first type of statistical factors, and the first type of statistical factors include the number of target image frames, by judging whether the number of target image frames is within a third preset numerical range, it is determined whether the first type of statistical factors meet the corresponding foreign object detection conditions.

[0014] Therefore, by setting the above foreign matter detection conditions, it is possible to use statistical factors to determine the foreign matter situation of the target video, and then determine whether the target video contains foreign matter.

[0015] Among them, the preset proportion range is less than the proportion threshold; the first preset value range is greater than the first threshold, and the first threshold is less than the score threshold; the second preset value range is greater than the second threshold; and the third preset value range is greater than the third threshold.

[0016] Therefore, by setting the above threshold, the foreign matter situation of the target video can be determined by comparing the threshold with the numerical value of the statistical factor.

[0017] Among them, several categories of statistical factors include first category statistical factors and second category statistical factors; based on several categories of statistical factors, determining the foreign object detection result of the target video includes: in response to both the first category statistical factors and the second category statistical factors satisfying the corresponding foreign object detection conditions, determining that the foreign object detection result of the target video does not contain foreign objects, or, in response to the first category statistical factors or the second category statistical factors satisfying the corresponding foreign object detection conditions, determining that the foreign object detection result of the target video does not contain foreign objects.

[0018] Therefore, two types of statistical factors can be set, and it can also be set that both types of statistical factors need to be satisfied or only one of them needs to be satisfied, that is, it can be determined that no foreign matter is contained, which can be flexibly adapted to different application needs.

[0019] Among them, the number of first-class statistical factors is at least one; wherein, in response to the first-class statistical factors or the second-class statistical factors satisfying the corresponding foreign body detection conditions, determining the foreign body detection result of the target video as not containing foreign objects includes: judging whether each first-class statistical factor satisfies the foreign body detection conditions corresponding to each first-class statistical factor; in response to each first-class statistical factor satisfying the foreign body detection conditions corresponding to each first-class statistical factor, determining the foreign body detection result of the target video as not containing foreign objects; in response to the existence of any first-class statistical factor not satisfying the corresponding foreign body detection conditions, judging whether the second-class statistical factor satisfies the foreign body detection conditions corresponding to the second-class statistical factor; in response to the second-class statistical factor satisfying the foreign body detection conditions corresponding to the second-class statistical factor, determining the foreign body detection result of the target video as not containing foreign objects.

[0020] Therefore, we can first determine whether the first type of statistical factors meet the corresponding foreign object detection conditions. If the first type of statistical factors do not meet the corresponding foreign object detection conditions, we can further determine whether the second type of statistical factors meet the corresponding foreign object detection conditions. This can achieve the foreign object detection results of the target video based on the two types of statistical factors.

[0021] Among them, obtaining the images to be detected corresponding to several target image frames in the target video includes: performing target detection on each image frame contained in the target video to obtain the target detection result of each image frame; based on the target detection result of each image frame, selecting several target image frames from all image frames, and intercepting the corresponding images to be detected from the target image frames.

[0022] Therefore, according to the target detection results of each image frame, the object to be detected is selected from the target video, and the image that needs to be detected for foreign object detection can be accurately selected to perform foreign object detection in a targeted manner.

[0023] Among them, at least one frame of target detection results includes a part detection frame and an object detection frame. Based on the target detection results of each image frame, several target image frames are selected from all image frames, and corresponding images to be detected are captured from the target image frames, including: in response to the existence of an object detection frame located inside the part detection frame in the image frame, the image frame is used as the target image frame; from the target image frame, the image corresponding to the object detection frame located inside the part detection frame is captured as the image to be detected.

[0024] Therefore, through the positional relationship between the part detection frame and the object detection frame, the image of the object held by the user's body part can be accurately located, so that foreign body detection can be performed in a targeted manner.

[0025] Among them, using the foreign matter detection model, each image to be detected is judged whether it contains foreign matter, and the foreign matter judgment score of each image to be detected is obtained, which includes: extracting features of the image to be detected to obtain image features; performing binary classification on the image features to obtain a first probability that the image to be detected contains foreign matter and a second probability that the image contains goods; and determining the foreign matter judgment score based on the first probability or the second probability.

[0026] Therefore, by performing feature extraction and binary classification on the image to be detected, it is possible to determine the probability that the image contains a commodity and the probability that the image contains a foreign object, thereby obtaining a foreign object judgment score that represents the probability that the image to be detected contains a foreign object.

[0027] The foreign body detection model is trained using sample images, which include product images and foreign body images.

[0028] Therefore, the foreign body detection model fully learns the features of product images and foreign body images, so that it can accurately distinguish product images from foreign body images.

[0029] The present application provides a foreign object detection device, including an acquisition module, a foreign object detection module, and a determination module. The acquisition module is used to acquire images to be detected corresponding to a plurality of target image frames in a target video, wherein the target video is acquired from a commodity placement area; the foreign object detection module is used to use a foreign object detection model to perform foreign object detection on each image to be detected, respectively, to obtain a foreign object judgment score of each image to be detected, wherein the foreign object judgment score represents the probability that the image to be detected contains a foreign object, and the foreign object is an object other than a commodity; the determination module is used to determine a foreign object detection result of the target video based at least on the foreign object judgment score of each image to be detected, wherein the foreign object detection result is whether the foreign object is contained or not contained.

[0030] The present application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement any of the above-mentioned foreign body detection methods.

[0031] The present application provides a computer-readable storage medium on which program instructions are stored. When the program instructions are executed by a processor, any of the above-mentioned foreign matter detection methods is implemented.

[0032] In the above scheme, a target video of the commodity placement area is obtained, and a foreign object detection model is used to perform foreign object detection on the image to be detected contained in the target video to obtain the probability that the image contains foreign objects, thereby determining whether the target video contains foreign objects.

[0033] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0035] Figure 1 It is a flow chart of an embodiment of a foreign body detection method of the present application;

[0036] Figure 2 This application Figure 1 A flowchart of another embodiment of step S130;

[0037] Figure 3 This application Figure 1 A flowchart of another embodiment of step S110;

[0038] Figure 4 is a schematic diagram of another embodiment of the foreign body detection method of the present application;

[0039] Figure 5 It is a schematic diagram of the framework of an embodiment of a foreign body detection device of the present application;

[0040] Figure 6 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application;

[0041] Figure 7 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0042] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0043] In the following description, for the purpose of explanation rather than limitation, specific details such as specific subsystem structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0044] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0045] See also Figure 1 , Figure 1 It is a flow chart of an embodiment of a foreign body detection method of the present application.

[0046] Specifically, the method may include:

[0047] Step S110: obtaining images to be detected corresponding to a plurality of target video frames in the target video.

[0048] The target video is obtained by collecting the product placement area. The product placement area can be an area set in a device, and the device can be a vending machine, a vending machine, a refrigerator for selling goods, etc. The foreign body detection method can determine whether a foreign body is placed in the above-mentioned device, and can help users determine the abnormal risk during the operation of the device.

[0049] The target video includes several target image frames, each of which corresponds to an image to be detected.

[0050] It is understandable that the image to be detected can be obtained from the target video frame and can be used as the object for foreign body detection. The image to be detected can contain an object, and foreign body detection can be used to determine whether the object is a foreign body or a commodity.

[0051] Step S120: using the foreign object detection model, respectively perform foreign object detection on each image to be detected to obtain a foreign object judgment score for each image to be detected.

[0052] Foreign matter is an object that is not a commodity. Foreign matter detection is used to determine whether the object contained in the image is a commodity or a foreign matter. Each image to be detected is processed separately to obtain a foreign matter judgment score for each image to be detected. The foreign matter judgment score can represent the probability that the image to be detected contains a foreign matter.

[0053] In some embodiments, performing foreign body detection may include: extracting features from the image to be detected to obtain image features, performing binary classification on the image features to obtain a first probability that the image to be detected contains foreign bodies and a second probability that the image to be detected contains goods. Based on the first probability or the second probability, determining a foreign body judgment score. The sum of the first probability and the second probability may be one.

[0054] It should be noted that the foreign body judgment score can be set so that a larger value indicates a more likely commodity, or can be set so that a larger value indicates a more likely foreign body.

[0055] In some embodiments, foreign body detection can be implemented using a foreign body detection model, which can be a neural network model trained using sample images, wherein the sample images include product images and foreign body images. Among them, foreign body images can include images manually collected in the device, images crawled from the Internet, and potential foreign body images contained in academic data sets. By learning foreign body images and product images together, the foreign body detection model can learn the knowledge of products and foreign bodies, so that the foreign body detection model can accurately distinguish whether the image contains foreign bodies or products, thereby achieving accurate foreign body detection.

[0056] In a specific application scenario, the foreign object detection model uses a deep residual (ResNet34) network.

[0057] Step S130: determining a foreign matter detection result of the target video based at least on the foreign matter judgment score of each image to be detected.

[0058] Among them, the foreign object detection result can be that the foreign object is contained or not contained, which can indicate whether the target video contains foreign objects.

[0059] It should be noted that the foreign object judgment score can represent the probability that the image to be detected contains foreign objects. By using the foreign object judgment scores corresponding to each target image frame contained in the target video, it is possible to determine whether the target video contains foreign objects and realize foreign object detection in the target video.

[0060] Of course, in some embodiments, other information and the foreign object judgment score may be used together to determine the foreign object detection result of the target video.

[0061] In the above scheme, a target video of the commodity placement area is obtained, and a foreign object detection model is used to perform foreign object detection on the image to be detected contained in the target video to obtain the probability that the image contains foreign objects, thereby determining whether the target video contains foreign objects.

[0062] In a specific application scenario, goods are placed in a refrigerator for selling goods. During the use of the refrigerator, in order to promptly identify whether a user has placed goods that are not for sale, placed personal belongings, or carried out malicious attacks by placing non-goods, a target video of the refrigerator is captured and foreign object detection is performed. The foreign object detection result of the target video is determined, and abnormal placement can be detected in time to prompt the user, thereby reducing the risks in the operation of the refrigerator.

[0063] See also Figure 2 , Figure 2 This application Figure 1 Schematic diagram of another embodiment of step S130 in FIG. 1 . Specifically, this step may include:

[0064] Step S231: Obtain several types of statistical factors.

[0065] The statistical factors may be statistical factors about the target video, may represent the characteristics of the target video, and may be used to determine whether there is a foreign object in the target video. At least one type of statistical factors is determined based on the foreign object judgment score of the image to be detected.

[0066] In some cases, one type of statistical factor may be provided to determine the foreign body detection result. In some cases, multiple types of statistical factors may be provided to determine the foreign body detection result.

[0067] Step S232: Determine the foreign body detection result of the target video based on several types of statistical factors.

[0068] Among them, various statistical factors can correspond to foreign matter detection conditions respectively, and the foreign matter detection result is determined to contain foreign matter or not contain foreign matter according to whether the statistical factors meet the corresponding foreign matter detection conditions.

[0069] Specifically, when any type of statistical factors meets the corresponding foreign body detection conditions, the foreign body detection result of the target video is that it does not contain foreign objects, or when all types of statistical factors meet the corresponding foreign body detection conditions, the foreign body detection result of the target video is that it does not contain foreign objects.

[0070] When a type of statistical factors is set, the foreign matter detection result can be determined based on whether the type of statistical factors meets the corresponding foreign matter detection conditions.

[0071] When multiple types of statistical factors are set, it can be set to determine that the target video does not contain foreign matter when any type of statistical factors meets the corresponding foreign matter detection conditions, or it can be set to determine that the target video contains foreign matter when all types of statistical factors meet the corresponding foreign matter detection conditions.

[0072] In some embodiments, the statistical factor may include at least one of the second statistical value of the foreign matter judgment score of all images to be detected and the target ratio of the number of suspected foreign matter to be detected to the total number of target image frames. In some cases, the statistical factor may also include at least one of the number of target image frames and the first statistical value of the commodity classification score of all images to be detected. Of course, in some cases, other statistical factors may be set according to actual application needs, and the above statistical factors may be divided into several categories.

[0073] In some embodiments, several types of statistical factors include at least one of a first type of statistical factor and a second type of statistical factor. Among them, the first type of statistical factor includes at least a second statistical value of the foreign body judgment score of all images to be detected. In some cases, the first type of statistical factor may also include at least one of the number of target image frames and the first statistical value of the commodity classification score of all images to be detected; the commodity classification score of the image to be detected represents the probability that the image to be detected contains commodities of a preset commodity category. Among them, the number of first type of statistical factors may be one or more. The second type of statistical factor includes a target ratio of the number of images to be detected suspected of containing foreign matter to the total number of target image frames, and the images to be detected suspected of containing foreign matter are determined by comparing the foreign body judgment score of the image to be detected with the score threshold. Of course, in some cases, the category division of the above statistical factors can also be adjusted according to actual application conditions.

[0074] In one implementation scenario, the second statistical value of the foreign body judgment scores of all the images to be detected may be the average value of the foreign body detection scores of all the images to be detected. Of course, in some cases, the second statistical value may not be limited to the above average value, and other forms of statistical values ​​may be used. The second statistical value can reflect the foreign body situation of the image frame in the target video, and can also be used to determine whether the target video contains foreign body.

[0075] In some embodiments, the foreign object detection condition corresponding to the second statistical value is that the second statistical value is within a first preset value range. In response to the plurality of types of statistical factors including the first type of statistical factors, and the first type of statistical factors including the second statistical value, it can be determined whether the first type of statistical factors meets the corresponding foreign object detection condition by judging whether the second statistical value is within the first preset value range.

[0076] In a specific application scenario, a larger value of the second statistical value indicates that it is more likely that no foreign matter is contained, and a foreign matter detection condition corresponding to the second statistical value is that the second statistical value is greater than the first threshold.

[0077] In one implementation scenario, the commodity classification score of the image to be detected is obtained by identifying the commodity category of the image to be detected. Commodity category identification can be used to identify which category of commodities among several preset commodity categories are contained in the image. The commodity classification score can characterize the probability that the image to be detected contains commodities of a preset commodity category. The first statistical value of the commodity classification scores of all images to be detected can be the average value of the commodity classification scores of all images to be detected. Of course, in some cases, the first statistical value may not be limited to the above average value, and other forms of statistical values ​​may be used. The first statistical value can reflect the situation that the image frame in the target video contains commodities, and can also reflect to a certain extent whether the target video contains foreign matter.

[0078] Furthermore, the commodity category recognition of the image to be detected may specifically include: extracting features of the image to be detected, comparing the extracted image features with preset commodity features contained in a commodity feature library, wherein the commodity feature library contains preset commodity features corresponding to each preset commodity category, and determining the similarity between the image features and each preset commodity feature. The commodity classification score of the image to be detected is determined based on the above similarity. Specifically, the commodity classification score of the image to be detected can be determined based on the highest similarity, indicating the probability that the image to be detected contains a commodity of a preset commodity category corresponding to the preset commodity feature.

[0079] In some embodiments, the foreign object detection condition corresponding to the first statistical value is that the first statistical value is within a second preset value range. In response to the plurality of types of statistical factors including the first type of statistical factors, and the first type of statistical factors including the first statistical value, it can be determined whether the first type of statistical factors meets the corresponding foreign object detection condition by judging whether the first statistical value is within the second preset value range.

[0080] In a specific application scenario, a larger first statistical value indicates a higher probability that no foreign matter is contained, and a foreign matter detection condition corresponding to the first statistical value is that the first statistical value is greater than a second threshold.

[0081] In one implementation scenario, the number of target image frames can represent the number of images to be detected for foreign object detection. It can represent the length of the portion of the target video where the object to be detected exists, and can be used to help determine whether there is a foreign object in the target video. For example, if the number of target image frames is small, some statistical values ​​may have certain errors, such as average values, due to the small number of statistical samples. Therefore, the number of target image frames can help judge the credibility of the statistical values.

[0082] In some embodiments, the foreign object detection condition corresponding to the number of target image frames is that the number of target image frames is within a third preset value range. In response to the plurality of types of statistical factors including the first type of statistical factors, and the first type of statistical factors including the number of target image frames, it can be determined whether the first type of statistical factors meets the corresponding foreign object detection condition by judging whether the number of target image frames is within the third preset value range.

[0083] In a specific application scenario, the foreign object detection condition corresponding to the number of target image frames is that the number of target image frames is greater than a third threshold.

[0084] In one implementation scenario, a score threshold can be set for the foreign object judgment score, and the score threshold can be used to distinguish whether the image is suspected of containing foreign objects. The target ratio of the number of images suspected of containing foreign objects to be detected to the total number of target image frames can reflect the foreign object situation in the target video.

[0085] In some embodiments, the foreign body detection condition corresponding to the target ratio is that the target ratio is within a preset ratio range. In response to the plurality of statistical factors including the second statistical factor, and the second statistical factor being the target ratio, it can be determined whether the second statistical factor satisfies the corresponding foreign body detection condition by judging whether the target ratio is within the preset ratio range.

[0086] In a specific application scenario, a score threshold is set for the foreign body judgment score, and the image to be detected whose foreign body judgment score is less than the score threshold can be considered to be suspected of containing foreign bodies. It can be understood that the number of target image frames is consistent with the number of images to be detected, and the target ratio of the number of images to be detected suspected of containing foreign bodies to the total number of target image frames is counted. Among them, the foreign body detection condition corresponding to the target ratio can be a preset ratio range that is less than the ratio threshold.

[0087] In some embodiments, the several types of statistical factors include a first type of statistical factors. When the first type of statistical factors satisfy corresponding foreign object detection conditions, the foreign object detection result of the target video is determined to contain no foreign objects.

[0088] In a specific application scenario, the plurality of statistical factors include a first statistical factor, the first statistical factor includes a second statistical value of the foreign body judgment score of all images to be detected, and when the second statistical value is greater than a first threshold, the foreign body detection result of the target video is determined to contain no foreign body. When the second statistical value is less than or equal to the first threshold, the foreign body detection result of the target video is determined to contain foreign body.

[0089] In some implementation scenarios, there are multiple first-category statistical factors, and each first-category statistical factor corresponds to its own foreign body detection condition. It can be set that when each first-category statistical factor satisfies the corresponding foreign body detection condition, it can be considered that the first-category statistical factor satisfies the corresponding foreign body detection condition. If there are one or more first-category statistical factors that do not satisfy their corresponding foreign body detection conditions, it can be considered that the first-category statistical factor does not satisfy the corresponding foreign body detection condition.

[0090] In a specific application scenario, the plurality of statistical factors include a first statistical factor, and the first statistical factor includes a second statistical value of the foreign body judgment score of all images to be detected and the number of target image frames. When the second statistical value is greater than the first threshold value and the number of target image frames is greater than the third threshold value, it is determined that the foreign body detection result of the target video does not contain foreign bodies.

[0091] In a specific application scenario, the plurality of statistical factors include a first statistical factor, and the first statistical factor includes a second statistical value of the foreign body judgment score of all images to be detected and a first statistical value of the commodity classification score of all images to be detected. When the second statistical value is greater than the first threshold value and the first statistical value is greater than the second threshold value, it is determined that the foreign body detection result of the target video does not contain foreign bodies.

[0092] In a specific application scenario, several types of statistical factors include a first type of statistical factors, and the first type of statistical factors include a second statistical value of the foreign body judgment scores of all images to be detected, the number of target image frames, and a first statistical value of the commodity classification scores of all images to be detected. When the second statistical value is greater than the first threshold, the number of target image frames is greater than the third threshold, and the first statistical value is greater than the second threshold, it is determined that the foreign body detection result of the target video does not contain foreign bodies.

[0093] It is understandable that the first type of statistical factors may have different settings, which will not be illustrated one by one here.

[0094] In some embodiments, the several types of statistical factors include a second type of statistical factors. When the second type of statistical factors satisfy corresponding foreign object detection conditions, the foreign object detection result of the target video is determined to contain no foreign objects.

[0095] In a specific application scenario, the plurality of statistical factors include a second statistical factor, and the second statistical factor includes a target ratio of the number of images to be detected that are suspected of containing foreign objects to the total number of target image frames. Statistics are performed based on the foreign object judgment scores of each image to be detected to obtain a target ratio, and when the target ratio is less than a ratio threshold, the foreign object detection result of the target video is determined to contain no foreign objects.

[0096] In some embodiments, the several types of statistical factors include a first type of statistical factors and a second type of statistical factors. Determining the foreign object detection result of the target video may include: in response to both the first type of statistical factors and the second type of statistical factors satisfying corresponding foreign object detection conditions, determining that the foreign object detection result of the target video does not contain foreign objects, or, in response to the first type of statistical factors or the second type of statistical factors satisfying corresponding foreign object detection conditions, determining that the foreign object detection result of the target video does not contain foreign objects.

[0097] Exemplarily, the first type of statistical factors includes the second statistical value, and the second type of statistical factors includes the target ratio; or, the first type of statistical factors includes the second statistical value and the first statistical value, and the second type of statistical factors includes the target ratio; or, the first type of statistical factors includes the second statistical value, the first statistical value, and the number of target image frames, and the second type of statistical factors includes the target ratio.

[0098] In a specific application scenario, several types of statistical factors include first type statistical factors and second type statistical factors. The first type statistical factors include the second statistical value, the first statistical value, and the number of target image frames, and the second type statistical factors include the target ratio. When the second statistical value is greater than the first threshold, the number of target image frames is greater than the third threshold, and the first statistical value is greater than the second threshold, and the target ratio is less than the ratio threshold, it is determined that the foreign body detection result of the target video does not contain foreign bodies.

[0099] In one implementation scenario, the several types of statistical factors include first-type statistical factors and second-type statistical factors. Determining that the foreign object detection result of the target video does not contain foreign objects may include: judging whether each first-type statistical factor satisfies the foreign object detection condition corresponding to each first-type statistical factor, and in response to each first-type statistical factor satisfying the foreign object detection condition corresponding to each first-type statistical factor, determining that the foreign object detection result of the target video does not contain foreign objects; in response to the existence of any first-type statistical factor that does not satisfy the corresponding foreign object detection condition, judging whether the second-type statistical factor satisfies the foreign object detection condition corresponding to the second-type statistical factor; in response to the second-type statistical factor satisfying the foreign object detection condition corresponding to the second-type statistical factor, determining that the foreign object detection result of the target video does not contain foreign objects.

[0100] In a specific application scenario, several types of statistical factors include first-type statistical factors and second-type statistical factors. The first-type statistical factors include the second statistical value, the first statistical value, and the number of target image frames, and the second-type statistical factors include the target ratio. It is determined whether the second statistical value is greater than the first threshold, whether the first statistical value is greater than the second threshold, and whether the number of target image frames is greater than the third threshold. If all three are satisfied, the foreign body detection result of the target video is determined to contain no foreign matter; otherwise, it is determined whether the target ratio is less than the ratio threshold. If it is satisfied, the foreign body detection result of the target video is determined to contain no foreign matter. If it is not satisfied, the foreign body detection result of the target video is determined to contain foreign matter. Among them, the first threshold and the ratio threshold are both thresholds set for the foreign body judgment score, and the first threshold can be greater than the ratio threshold.

[0101] Of course, in some cases, it is also possible to first determine whether the second type of statistical factors meet the corresponding foreign body detection conditions. If the second type of statistical factors do not meet the corresponding foreign body detection conditions, then determine whether the first type of statistical factors meet the corresponding foreign body detection conditions.

[0102] See also Figure 3 , Figure 3 This application Figure 1 Schematic diagram of another embodiment of step S110 in FIG. 1 . Specifically, this step may include:

[0103] Step S311: performing target detection on each image frame included in the target video to obtain a target detection result for each image frame.

[0104] Among them, target detection is used to detect parts and objects in an image frame. The target detection result of the image frame can represent the detection result of the parts and objects in the image frame.

[0105] In some embodiments, the target video may be a video of a user's body part entering the target area once and leaving the target area. For example, the target video may be a video of a user's hand entering a refrigerator once and leaving the refrigerator. The target video may be captured from a long video of a product placement area.

[0106] Step S312: Based on the target detection results of each image frame, a number of target image frames are selected from all the image frames, and corresponding images to be detected are captured from the target image frames.

[0107] It is understandable that the purpose of foreign object detection is actually to detect whether a foreign object has been placed in a device where goods are placed, and placing a foreign object is usually done by the user. Generally speaking, the items held by a user's body parts, such as hands, are goods. By detecting whether the objects held by the user's body parts are goods or foreign objects, it can be determined whether the device has a foreign object placed by the user. Then the object of foreign object detection is the object within the range of the user's body parts.

[0108] Among all the image frames of the target video, there may be image frames that do not contain the object of foreign body detection, so they can be excluded as target image frames. Exemplarily, the image does not contain the user's body parts, or there is no object, etc. The target detection result may include the position information of the parts and the position information of the object in the image frame. According to the part position information and the object position information represented by the target detection result, the target image frame can be selected, and the image to be detected can be intercepted according to the position information, and the image to be detected contains the object of foreign body detection.

[0109] Further, the target detection result of at least one frame includes part position information and object position information. The part position information is a part detection frame, and the object position information is an object detection frame. Selecting the target image frame and intercepting the image to be detected may include: selecting the target image frame and intercepting the image to be detected based on the positional relationship between the part and the object.

[0110] Specifically, selecting a target image frame and intercepting an image to be detected may include: in response to the presence of an object detection frame located inside a part detection frame in an image frame, taking the image frame as a target image frame, and intercepting an image corresponding to the object detection frame located inside the part detection frame from the target image frame as the image to be detected.

[0111] In a specific application scenario, in response to the existence of an object detection frame and a hand detection frame in an image frame, and the object detection frame is located inside the hand detection frame, the image frame is used as the target image frame, and the image corresponding to the object detection frame located inside the hand detection frame is captured as the image to be detected, so that the image of the object that needs to be detected as a foreign object can be located.

[0112] In some embodiments, there may be multiple object detection frames located inside the part detection frame in the same image frame. Then each object detection frame located inside the part detection frame can capture a sub-image, and the multiple sub-images serve as the images to be detected corresponding to the target image frame. The foreign body judgment score of the image to be detected can be obtained based on the foreign body judgment scores of each sub-image. For example, the sub-image with the lowest foreign body judgment score can be selected as the foreign body judgment score of the image to be detected corresponding to the target image frame. The same applies to the commodity classification score.

[0113] Alternatively, in some cases, there may be multiple object detection frames located inside the part detection frame in the same image frame, so each object detection frame located inside the part detection frame can be correspondingly captured into an image to be detected. At this time, the number of images to be detected may be inconsistent with the number of target image frames. When counting the target ratio, the image to be detected that is suspected of containing foreign objects can be compared with the number of images to be detected or the number of target image frames to obtain the target ratio.

[0114] See also Figure 4 , Figure 4 It is a schematic diagram of another embodiment of the foreign body detection method of the present application.

[0115] In this embodiment, a target detection model, a foreign body detection model, and a commodity classification model are constructed respectively. The foreign body detection model is trained using sample images, and the sample images include commodity images and foreign body images. The model is trained using the stochastic gradient descent method. The above model can use a deep residual (ResNet34) network.

[0116] Among them, the target detection model is used to detect parts and objects in the image. Based on the target detection results, the image to be detected can be obtained. The image to be detected is input into the foreign body detection model and the product classification model respectively to obtain the foreign body judgment score and the product classification score respectively.

[0117] The foreign body detection model can perform binary classification on the image to obtain the probability corresponding to the product category and the probability corresponding to the foreign body category. The foreign body judgment score can be the probability corresponding to the product category output by the foreign body detection model.

[0118] In a specific application scenario, for the image frames contained in the refrigerator commodity transaction video, the target detection model is input, the target image frame is selected and the image to be detected is intercepted, and the foreign object detection model and the commodity classification model are used to obtain the foreign object judgment score and commodity classification score of each image to be detected. It is judged whether the following three indicators are met. If they are met at the same time, it is considered to be a normal commodity: the average value (Retrieval score mean) of the commodity classification score of all images to be detected is greater than 0.95, the average value (Ood score mean) of the foreign object judgment score of all images to be detected is greater than 13, and the number of target image frames is greater than 7. The above thresholds are only examples and can be adjusted according to actual applications.

[0119] If the refrigerator commodity transaction video does not meet the above three indicators, it is judged whether it meets the following indicators: the target ratio is less than the ratio threshold, and the target ratio is the ratio of the number of foreign matter judgment scores of the images to be detected that are less than the score threshold to the total number of target image frames. If it meets the requirements, it is considered to be a normal transaction, otherwise it is considered to contain foreign matter.

[0120] In some embodiments, after determining that the target video contains foreign matter, an alarm signal may be issued to warn the user.

[0121] See also Figure 5 , Figure 5 It is a schematic diagram of the framework of an embodiment of a foreign body detection device of the present application.

[0122] In this embodiment, the foreign object detection device 50 includes an acquisition module 51, a foreign object detection module 52 and a determination module 53. The acquisition module 51 is used to acquire images to be detected corresponding to several target image frames in the target video, and the target video is collected from the commodity placement area; the foreign object detection module 52 is used to use the foreign object detection model to perform foreign object detection on each image to be detected, and obtain a foreign object judgment score for each image to be detected. The foreign object judgment score represents the probability that the image to be detected contains a foreign object, and the foreign object is an object that is not a commodity; the determination module 53 is used to determine the foreign object detection result of the target video based on at least the foreign object judgment score of each image to be detected, and the foreign object detection result is whether it contains a foreign object or not.

[0123] See also Figure 6 , Figure 6 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application.

[0124] In this embodiment, the electronic device 60 includes a memory 61 and a processor 62, and the processor 62 is used to execute the program instructions stored in the memory 61 to implement the steps in any of the above foreign body detection method embodiments. In a specific implementation scenario, the electronic device 60 may include but is not limited to: computer equipment, electrical equipment, microcomputers, desktop computers, servers, and in addition, the electronic device 60 may also include mobile devices such as laptops and tablet computers, which are not limited here.

[0125] Specifically, the processor 62 is used to control itself and the memory 61 to implement the steps in any of the above-mentioned summary generation method embodiments. The processor 62 can also be called a CPU (Central Processing Unit). The processor 62 may be an integrated circuit chip with signal processing capabilities. The processor 62 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 62 can be implemented by an integrated circuit chip.

[0126] See also Figure 7 , Figure 7 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application.

[0127] The computer-readable storage medium 70 provided in this embodiment stores program instructions 71 that can be executed by a processor. When the program instructions 71 are executed by the processor, they are used to implement any of the above-mentioned foreign matter detection methods.

[0128] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0129] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another subsystem, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0130] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A foreign body detection method, characterized in that: The method comprises: Obtaining images to be detected corresponding to a plurality of target image frames in a target video, wherein the target video is acquired from a commodity placement area; Using the foreign object detection model, each of the images to be detected is subjected to foreign object detection to obtain a foreign object judgment score of each of the images to be detected, wherein the foreign object judgment score represents the probability that the image to be detected contains a foreign object, and the foreign object is an object other than a commodity; Obtaining a first type of statistical factor and a second type of statistical factor; wherein the first type of statistical factor includes the number of the target image frames, a first statistical value of the commodity classification scores of all the images to be detected, and a second statistical value of the foreign body judgment scores of all the images to be detected, the commodity classification scores representing the probability that the images to be detected contain commodities of a preset commodity category; the second type of statistical factor is the target ratio of the number of images to be detected suspected of containing foreign matter to the total number of the target image frames, the images to be detected suspected of containing foreign matter are determined by comparing the foreign body judgment scores of the images to be detected with the score threshold; Based on the first type of statistical factors and the second type of statistical factors, a foreign object detection result of the target video is determined, and the foreign object detection result is including foreign objects or not including foreign objects.

2. The method according to claim 1, characterized in that In response to the second-type statistical factor being the target ratio, determining whether the second-type statistical factor satisfies a corresponding foreign object detection condition by judging whether the target ratio is within a preset ratio range; In response to the first type of statistical factors including the second statistical value, determining whether the first type of statistical factors meets the corresponding foreign body detection condition by judging whether the second statistical value is within a first preset value range; In response to the first type of statistical factors including the first statistical value, determining whether the first type of statistical factors meets the corresponding foreign object detection condition by judging whether the first statistical value is within a second preset value range; In response to the first type of statistical factors including the number of the target image frames, it is determined whether the first type of statistical factors meets the corresponding foreign object detection condition by judging whether the number of the target image frames is within a third preset value range.

3. The method according to claim 1, characterized in that The determining the foreign body detection result of the target video based on the first type of statistical factors and the second type of statistical factors includes: In response to both the first-category statistical factor and the second-category statistical factor satisfying corresponding foreign object detection conditions, it is determined that the foreign object detection result of the target video does not contain foreign objects.

4. The method according to claim 1, characterized in that The determining the foreign body detection result of the target video based on the first type of statistical factors and the second type of statistical factors includes: Determining whether each of the first-category statistical factors satisfies a foreign body detection condition corresponding to each of the first-category statistical factors; In response to each of the first-category statistical factors satisfying the foreign object detection conditions corresponding to each of the first-category statistical factors, determining that the foreign object detection result of the target video does not contain a foreign object; In response to any of the first-category statistical factors not satisfying the corresponding foreign body detection condition, determining whether the second-category statistical factors satisfy the foreign body detection condition corresponding to the second-category statistics; In response to the second-type statistical factor satisfying the foreign object detection condition corresponding to the second-type statistics, it is determined that the foreign object detection result of the target video does not contain foreign objects.

5. The method according to claim 1, characterized in that The step of obtaining the images to be detected corresponding to the plurality of target image frames in the target video comprises: Performing target detection on each image frame included in the target video respectively to obtain a target detection result for each image frame; Based on the target detection results of each of the image frames, the several target image frames are selected from all the image frames, and the corresponding images to be detected are intercepted from the target image frames.

6. The method according to claim 5, characterized in that At least one frame of the target detection result includes a part detection frame and an object detection frame, and the target detection results of each image frame are selected from all the image frames, and the corresponding image to be detected is intercepted from the target image frame, including: In response to the object detection frame being located inside the part detection frame in the image frame, taking the image frame as the target image frame; From the target image frame, an image corresponding to the object detection frame located inside the part detection frame is intercepted as the image to be detected.

7. The method according to any one of claims 1 to 6, characterized in that: The foreign body detection model is used to perform foreign body detection on each of the images to be detected, and the foreign body judgment score of each of the images to be detected is obtained, which includes: Extracting features from the image to be detected to obtain image features; Performing binary classification on the image features to obtain a first probability that the image to be detected contains a foreign object and a second probability that the image to be detected contains a commodity; Determining the foreign matter judgment score based on the first probability or the second probability; And / or, the foreign object detection model is trained using sample images, and the sample images include product images and foreign object images.

8. A foreign body detection device, characterized in that: include: An acquisition module is used to acquire images to be detected corresponding to a plurality of target image frames in a target video, wherein the target video is acquired from a commodity placement area; A foreign body detection module, used to use a foreign body detection model to perform foreign body detection on each of the images to be detected, and obtain a foreign body judgment score for each of the images to be detected, wherein the foreign body judgment score represents the probability that the image to be detected contains a foreign body, and the foreign body is a non-commodity object; A determination module, used to obtain the first type of statistical factors and the second type of statistical factors; Based on the first category of statistical factors and the second category of statistical factors, the foreign body detection result of the target video is determined; wherein, the first category of statistical factors includes the number of the target image frames, a first statistical value of the commodity classification scores of all the images to be detected, and a second statistical value of the foreign body judgment scores of all the images to be detected, and the commodity classification score represents the probability that the image to be detected contains commodities of a preset commodity category; the second category of statistical factors is the target ratio of the number of images to be detected suspected of containing foreign objects to the total number of the target image frames, and the images to be detected suspected of containing foreign objects are determined by comparing the foreign body judgment scores of the images to be detected with the score threshold; the foreign body detection result is that foreign matter is contained or not contained.

9. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores program instructions, and when the program instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images