Goods state detection device and method

Abstract

The embodiment of the invention provides a goods state detection device and method, and the device comprises a first detection unit which extracts edge features from a first image, generates a first edge feature image, extracts edge features from a second image, and generates a second edge feature image; the first determination unit is used for calculating a first difference image between the second edge feature image and the first edge feature image, and determining a goods state change area according to the first difference image; and the second determining unit is used for determining the goods state change type in the goods state change area.

Description

Technical Field

[0001] The present application relates to the field of image processing. Background Art

[0002] Traditional retail supermarkets and convenience stores require full-time staff, which results in high labor costs. In related technologies, staff are required to check whether the goods on the shelves are replenished, out of stock, or moved by customers. The commonly used change detection method is to compare images taken at different times, find the changed parts in the image, and then make a judgment. However, this change detection method is usually applied to outdoor scenes, such as satellite remote sensing. This is because in outdoor scenes, the image pairs are taken from satellites, and most of the content in the image pairs has changed. As long as the changed area is detected, it is sufficient. For indoor retail scenes, the image pairs are taken by fixed surveillance cameras, and most of the content in the image pairs has not changed. There are a large number of dense goods with similar textures and shapes on the shelves, which makes the above change detection method more difficult to implement.

[0003] It should be noted that the above introduction to the technical background is only for the convenience of providing a clear and complete description of the technical solutions of the present application and for the convenience of understanding by those skilled in the art. It cannot be considered that the above technical solutions are well known to those skilled in the art simply because they are described in the background technology section of this document. Summary of the invention

[0004] In response to at least one of the above problems, an embodiment of the present application provides a device and method for detecting the status of goods.

[0005] According to a first aspect of an embodiment of the present application, a device for detecting a state of goods is provided, the device comprising:

[0006] A first detection unit, which extracts edge features from the first image to generate a first edge feature image, and extracts edge features from the second image to generate a second edge feature image;

[0007] A first determining unit, which calculates a first difference image between the second edge feature image and the first edge feature image, and determines a product state change area according to the first difference image;

[0008] The second determining unit determines a type of a change in the state of the product in the state of the product change area.

[0009] According to a second aspect of an embodiment of the present application, a method for detecting a state of goods is provided, the method comprising:

[0010] Extract edge features from the first image to generate a first edge feature image, and extract edge features from the second image to generate a second edge feature image;

[0011] Calculating a first difference image between the second edge feature image and the first edge feature image, and determining a product state change area according to the first difference image;

[0012] Determine the type of change in the status of the product in the product status change area.

[0013] One of the beneficial effects of the embodiments of the present application is that the area where the state of goods changes and the type of the state of goods change are determined based on the difference image of the edge images extracted from different images. Therefore, the automatic determination of the type of state of goods change can be achieved based only on the image acquired by the monocular camera, and the determination based on the difference image can improve the accuracy of the determination and is relatively low in cost.

[0014] With reference to the following description and accompanying drawings, the specific embodiments of the present application are disclosed in detail, indicating the way in which the principles of the present application can be adopted. It should be understood that the embodiments of the present application are not limited in scope. Within the scope of the spirit and clauses of the appended claims, the embodiments of the present application include many changes, modifications and equivalents.

[0015] Features described and / or illustrated with respect to one embodiment may be used in the same or similar manner in one or more other embodiments, combined with features in other embodiments, or substituted for features in other embodiments.

[0016] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, integers, steps or components, but does not exclude the presence or addition of one or more other features, integers, steps or components. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The included drawings are used to provide a further understanding of the embodiments of the present application, which constitute a part of the specification, are used to illustrate the implementation methods of the present application, and together with the text description, explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0018] Figure 1 is a schematic diagram of a goods status detection device according to an embodiment of the present application;

[0019] Figure 2 is a schematic diagram of a first difference image used in an embodiment of the present application to determine a product status change area;

[0020] Figure 3 is a schematic diagram of a depth image and a second difference image according to an embodiment of the present application;

[0021] Figure 4 is a schematic diagram of a neural network structure of an embodiment of the present application;

[0022] Figure 5 is a schematic diagram of an electronic device according to an embodiment of the present application;

[0023] Figure 6 is a schematic block diagram of the system structure of the electronic device according to an embodiment of the present application;

[0024] Figure 7 is a schematic diagram of a method for detecting a product status according to an embodiment of the present application;

[0025] Figure 8 It is a schematic diagram of a method for detecting the status of goods according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] In the embodiments of the present application, the terms "first", "second", etc. are used to distinguish different elements in terms of title, but do not indicate the spatial arrangement or temporal order of these elements, etc., and these elements should not be limited by these terms. The term "and / or" includes any one and all combinations of one or more of the associated listed terms. The terms "comprising", "including", "having", etc. refer to the presence of the stated features, elements, components or components, but do not exclude the presence or addition of one or more other features, elements, components or components.

[0027] In the embodiments of the present application, the singular forms "a", "the", etc. include plural forms and should be broadly understood as "a kind" or "a type" rather than being limited to the meaning of "one"; in addition, the term "said" should be understood to include both singular and plural forms, unless the context clearly indicates otherwise. In addition, the term "according to" should be understood as "at least in part according to...", and the term "based on" should be understood as "at least in part based on...", unless the context clearly indicates otherwise.

[0028] With reference to the accompanying drawings, the above and other features of the present application will become apparent through the following description. In the description and the accompanying drawings, specific embodiments of the present application are specifically disclosed, which show some embodiments in which the principles of the present application can be adopted. It should be understood that the present application is not limited to the described embodiments. On the contrary, the present application includes all modifications, variations and equivalents falling within the scope of the attached claims.

[0029] Embodiments of the first aspect

[0030] An embodiment of the present application provides a device for detecting the status of goods. Figure 1 Schematic diagram of a product status detection device according to an embodiment of the present application. Figure 1 As shown, the product status detection device 100 includes:

[0031] A first detection unit 101, which extracts edge features from the first image to generate a first edge feature image, and extracts edge features from the second image to generate a second edge feature image;

[0032] A first determining unit 102, which calculates a first difference image between the second edge feature image and the first edge feature image, and determines a product state change area according to the first difference image;

[0033] The second determining unit 103 determines the type of change in the state of the product in the state of the product change area.

[0034] In some embodiments, the first image and the second image may be two frames of images in a video stream acquired by the same camera, which may be acquired by a camera shooting from the area where the goods are located, for example, by a surveillance camera, which is a monocular RGB camera. After acquiring a frame of a real-time surveillance video, the image is used as the first image, and after a predetermined time, another frame of image is acquired from the surveillance video, and the other frame of image is used as the second image. The predetermined time may be one day, one hour, 12 hours, one week, etc., and the embodiments of the present application are not limited thereto. The first image and the second image may be two-dimensional color images (RGB) or two-dimensional grayscale images, and the embodiments of the present application are not limited thereto. The first image and the second image may be the same size or may be processed to be the same.

[0035] In some embodiments, the first detection unit 101 may use an edge detection operator to extract edge features from the first image to generate a first edge feature image, and extract edge features from the second image to generate a second edge feature image. The edge detection operator includes but is not limited to at least one of the following operators: Prewitt operator, Laplacian operator, Roberts crossover operator, Canny operator, Sobel operator, etc. The first edge feature image and the second edge feature image may be binary images, that is, the edge feature part is displayed in white or gray, and the other part is displayed in black. The edge feature parts in the first edge feature image and the second edge feature image may be displayed in the same color or different colors, and the embodiments of the present application are not limited thereto.

[0036] In some embodiments, the second image can be regarded as an image after the state of the goods has changed, and the first image is an image before the state of the goods has changed. The first determination unit 102 calculates the difference between the second edge feature image and the first edge feature image (the difference in pixel values ​​at corresponding positions) to obtain a first difference image (e.g., a binary image), and determines the area where the state of the goods has changed based on the first difference image, wherein if the area where the state of the goods has not changed in the image has the same pixels at the corresponding position as the first edge feature image, the pixel difference is 0, and it can be displayed as black in the first difference image; if the area where the state of the goods has changed in the image has the same pixels at the corresponding position as the first edge feature image, the pixel difference is not 0, and it can be displayed as white and / or gray in the first difference image. In other words, the pixel area where the pixel value in the first difference image is not 0 can be used as the area where the state of the goods has changed.

[0037] In some embodiments, after obtaining the first difference image, the first determination unit may further perform noise reduction processing and graphics closing operations on the first difference image to generate the product state change region. Figure 2 is a schematic diagram of determining a product status change area using a first difference image according to an embodiment of the present application, such as Figure 2 As shown in a, the non-black (pixel value is not 0) pixel part in the first difference image is the product status change area, but in order to avoid the influence of light and shadow, the first difference image can be subjected to denoising to obtain the image in b. The denoising includes removing the pixel area whose connected pixel area (the area of ​​the area composed of adjacent pixels) is less than the predetermined area in the first difference image. The removal operation can be regarded as setting the pixel value of the corresponding pixel area to 0. By performing a graphics closing operation on the denoised image, the image in c is obtained. The non-black (pixel value is not 0) pixel part in c is the product status change area. The principle of the closing operation is to first perform dilation and then erode, so that the holes inside the connected area can be removed, the holes are filled by dilation, and then eroded to restore the size of the connected area, so as to better segment the product status change area. For the closing operation, reference can be made to the relevant technology, and the embodiment of the present application is not limited thereto.

[0038] It can be seen from the above embodiments that the determined area where the state of the goods changes can be represented by the coordinates of the pixels whose pixel values ​​are not 0 in the first difference image or the first difference image after noise reduction processing and closing operation, or, the minimum border surrounding the pixels whose pixel values ​​are not 0 in the first difference image or the first difference image after noise reduction processing and closing operation can be determined, and the area where the state of the goods changes is represented by the coordinates of the minimum border, but the embodiments of the present application are not limited to this.

[0039] In some embodiments, after determining the area where the product status changes, the second determination unit 103 may also determine the type of product status change in the product status change area, and the type of product status change includes taking away (out of stock), adding (replenishment), and moving. The embodiment of the present application proposes three algorithms: a deep difference algorithm, an edge classification algorithm, and a neural network algorithm to determine the type of product status change in the product status change area. The above three algorithms can be implemented independently or combined with each other. The embodiment of the present application is not limited to this, and is described in detail below.

[0040] In some embodiments, regarding the depth difference algorithm, the second determination unit 103 includes: a difference determination module; the difference determination module generates a first depth image according to the first image, generates a second depth image according to the second image, calculates a second difference image between the second depth image and the first depth image, and determines the type of change in the state of the goods in the state change area of ​​the goods in the second difference image. Among them, since the first image and the second image are images acquired by a monocular RGB camera, the existing monocular depth estimation model, such as AdaBins, etc., can be used to acquire the first depth image and the second depth image (grayscale image, different grayscale values ​​reflect different depths), that is, the first image is input into the monocular depth estimation model to obtain the first depth image, and the second image is input into the monocular depth estimation model to obtain the second depth image. The implementation method of the monocular depth estimation model can refer to the relevant technology, and the embodiment of the present application is not limited thereto. The difference determination module calculates the difference between the second depth image and the first depth image (the difference in pixel values ​​at corresponding positions) to obtain a second difference image (grayscale image). In the second difference image, the product state change area determined by the first determination unit 102 (the same pixel point position or the aforementioned minimum border position) is found, and the product state change type is determined within the product state change area.

[0041] In some embodiments, the difference determination module also performs standardization processing on the product change area in the second difference image, and determines the type of change in the product state according to the depth difference corresponding to the pixels in the product change area after the standardization processing. The standardization processing includes: normalizing the depth difference corresponding to the pixel points in the product change area to a predetermined range, for example, normalizing the depth difference corresponding to the pixel points in the product change area to a range of -1 to 1 in proportion, that is, after the standardization processing, 0 corresponds to no change in the depth difference, a positive value indicates that the depth value becomes larger, which is equivalent to the original presence of the product, and the product is taken away after the change, causing the position to be farther from the camera, and a negative value indicates that the depth value becomes smaller, which is equivalent to the original absence of the product, and the addition of the product after the change causes the position to be closer to the camera, but the embodiments of the present application are not limited to this. Figure 3is a schematic diagram of a depth image and a second difference image according to an embodiment of the present application, Figure 3 Where a is the first depth image and the second depth image, Figure 3 b in the figure is the second difference image, and the position marked by a circle in the second difference image is the product state change area determined by the first determination unit. Figure 3 c in the figure is a standardized processed image obtained by standardizing the area where the product status changes.

[0042] In some embodiments, the difference determination module determines the type of change in the state of the goods according to the proportional relationship between the positive and negative values ​​of the depth difference in the goods change area after the standardized processing. For example, in the goods change area after the standardized processing, the number of depth differences (normalized) corresponding to each pixel greater than 0 N1 and the number of depth differences (normalized) less than 0 N2 are counted, and the type of change in the state of the goods is determined according to the proportional relationship between N1 and N2. For example, if N1 / N1+N2 is greater than a preset value (for example, 70%), it indicates that the type of change in the state of the goods is taken away, and if N2 / N1+N2 is greater than a preset value (for example, 70%), it indicates that the type of change in the state of the goods is added, and in other cases it is moved. The preset value can be determined as needed, and the embodiments of the present application are not limited thereto.

[0043] In some embodiments, the difference determination module performs binary clustering on the depth difference in the product change area after the standardization process, and determines the type of change in the product state according to the positive and negative relationship between the center points of different categories. For example, the difference determination module can use a clustering algorithm to perform binary clustering on the depth difference in the product change area after the standardization process, and the clustering algorithm includes but is not limited to the K-means algorithm. After the binary clustering, the depth difference in the product change area after the standardization process is divided into two categories, and the center points of different categories are calculated (for example, the average value of the depth difference within the category). If the center points of the two categories are both positive, the type of change in the product state is taken away, if the center points of the two categories are both negative, the type of change in the product state is added, and if the center points of the two categories are one positive and one negative, the type of change in the product state is moved.

[0044] In some embodiments, regarding the edge classification algorithm, the second determination unit 103 includes: an edge determination module; the edge determination module determines a first number of pixel points belonging to the edge features of the second image in the product state change region in the first difference image, and determines a second number of pixel points belonging to the edge features of the first image. In the first difference image, there are pixel points belonging to the edge features of the first image and pixel points belonging to the edge features of the second image in the product state change region. For example, the edge feature parts in the second edge feature image and the first edge feature image are represented by white and gray respectively. In the product state change region of the first difference image, the number of pixel points of the gray line is the second number, and the number of pixel points of the white line is the first number.

[0045] In some embodiments, the edge determination module determines the type of change in the state of the goods in the state of the goods change area according to the relationship between the first quantity, the second quantity and the threshold value. For example, when the first quantity / (first quantity+second quantity) is greater than the threshold value, it means that the first quantity accounts for a larger proportion, and the type of change in the state of the goods is added. When the second quantity / (first quantity+second quantity) is greater than the threshold value, it means that the second quantity accounts for a larger proportion, and the type of change in the state of the goods is taken away. In other cases, the type of change in the state of the goods is moved. The threshold value can be determined as needed, for example, set to 0.6, but the embodiments of the present application are not limited thereto.

[0046] In some embodiments, the depth difference algorithm and the edge classification algorithm can be implemented separately or in combination. For example, when implemented in combination (hereinafter referred to as a hybrid algorithm), the edge determination module first determines whether the type of change in the state of the goods in the state change area is removal or addition. When the type of change in the state of the goods is not removal or addition, the difference determination module determines the type of change in the state of the goods. In this way, the accuracy of detecting the type of change in the state of the goods can be further improved.

[0047] In some embodiments, regarding the neural network algorithm, the second determination unit 103 includes: a neural network determination module; the neural network determination module uses the same neural network to extract a first feature and a second feature from the area where the state of the goods changes in the first image and the second image, calculates a feature difference between the first feature and the second feature, and determines the type of change in the state of the goods based on the feature difference. Figure 4 is a schematic diagram of the neural network structure of an embodiment of the present application, such as Figure 4As shown, after the first determination unit determines the area where the state of the goods changes, the first change area image I1 and the second change area image I2 are segmented from the original first image and the second image according to the position of the area where the state of the goods changes, and the first change area image and the second change area image are respectively input into the same convolutional neural network (sharing weights, structures, and parameters are the same), and the outputs of the two convolutional neural networks are the first feature and the second feature, respectively. The feature difference between the first feature and the second feature is calculated, and the feature difference is input into the fully connected network, and the output of the fully connected network is the type of change in the state of the goods. The convolutional neural network and the fully connected network can be trained based on the pre-collected image pairs and the marked types of change in the state of the goods as training data pairs, and the embodiments of the present application are not limited to this.

[0048] In some embodiments, the hybrid algorithm and the neural network algorithm can be implemented separately or in combination. When implemented in combination, the type of change in the state of the goods can be determined according to the hybrid algorithm and the neural network algorithm respectively, and then a comprehensive judgment is made according to the determination results of the two. For example, the first confidence of the type of change in the state of the goods determined by the hybrid algorithm and the second confidence of the type of change in the state of the goods determined by the neural network algorithm are determined respectively. If the first confidence is greater than the second confidence, the type of change in the state of the goods is determined to be the type of change in the state of the goods determined by the hybrid algorithm. If the first confidence is less than or equal to the second confidence, the type of change in the state of the goods is determined to be the type of change in the state of the goods determined by the neural network algorithm. Wherein, the first confidence can be determined according to the ratio value calculated in the deep difference algorithm or the edge classification algorithm (for example, the calculated ratio value is used as the first confidence, or the first confidence corresponding to the ratio value corresponding to different types is pre-determined according to the positive correlation between the ratio value and the first confidence), and the second confidence can be determined according to the probabilities of different types output by the neural network model (for example, the probabilities of each type output by the neural network model are used as the second confidence).

[0049] It is worth noting that the above only describes the components or modules related to the present application, but the present application is not limited thereto. The product status detection device 100 may also include other components or modules, and the specific contents of these components or modules may refer to the relevant technology.

[0050] To keep it simple, Figure 1 The connection relationship or signal direction between various components or modules is only exemplified, but it should be clear to those skilled in the art that various related technologies such as bus connection can be used. The above-mentioned components or modules can be implemented by hardware facilities such as processors and memories; the embodiments of the present application are not limited to this.

[0051] The above embodiments are merely exemplary of the embodiments of the present application, but the present application is not limited thereto, and appropriate modifications may be made based on the above embodiments. For example, the above embodiments may be used alone, or one or more of the above embodiments may be combined.

[0052] It can be seen from the above embodiments that the area where the state of goods changes and the type of the state of goods change are determined based on the difference image of the edge images extracted from different images. Therefore, the automatic determination of the type of state of goods change can be achieved based only on the image acquired by the monocular camera, and the determination based on the difference image can improve the accuracy of the determination and is relatively low in cost.

[0053] Embodiments of the second aspect

[0054] The embodiment of the present application also provides an electronic device, Figure 5 A schematic diagram of an electronic device according to an embodiment of the present application. Figure 5 As shown, the electronic device 500 includes a product status detection device 501. The structure and function of the product status detection device 501 are the same as those of the product status detection device 100 in the embodiment of the first aspect. For specific contents, please refer to the description in the embodiment of the first aspect, and will not be repeated here.

[0055] Figure 6 1 is a schematic block diagram of the system structure of the electronic device according to the embodiment of the present application. Figure 6 As shown, the electronic device 600 may include a processor 601 and a memory 602; the memory 602 is coupled to the processor 601. This figure is exemplary; other types of structures may also be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0056] In some embodiments, processor 601 includes at least one of a central processing unit (CPU) and a graphics processing unit (GPU).

[0057] like Figure 6 As shown, the electronic device 600 may further include: an input device 603, a display 604, and a power supply 605. In one embodiment, the function of the product status detection device may be integrated into the processor 601. The processor 601 may be configured to: extract edge features from the first image to generate a first edge feature image, extract edge features from the second image to generate a second edge feature image; calculate a first difference image between the second edge feature image and the first edge feature image, determine a product status change area according to the first difference image; determine the type of product status change in the product status change area. For other embodiments, specific reference may be made to the embodiments of the first aspect, which will not be described in detail here.

[0058] In another embodiment, the product status detection device may be configured separately from the processor 601 . For example, the product status detection device may be configured as a chip connected to the processor 601 , and the function of the product status detection device may be realized through the control of the processor 601 .

[0059] In this embodiment, the electronic device 600 does not necessarily include Figure 6 All parts shown in .

[0060] like Figure 6 As shown, the processor 601 is sometimes also referred to as a controller or an operation control, and may include a microprocessor or other processor devices and / or logic devices. The processor 601 receives inputs and controls the operations of various components of the electronic device 600 .

[0061] The memory 602 may be, for example, a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory or other suitable devices. The processor 601 may execute the program stored in the memory 602 to implement information storage or processing. The functions of other components are similar to those of the prior art and are not described in detail herein. The components of the electronic device 600 may be implemented by dedicated hardware, firmware, software or a combination thereof without departing from the scope of the present application.

[0062] It can be seen from the above embodiments that the area where the state of goods changes and the type of the state of goods change are determined based on the difference image of the edge images extracted from different images. Therefore, the automatic determination of the type of state of goods change can be achieved based only on the image acquired by the monocular camera, and the determination based on the difference image can improve the accuracy of the determination and is relatively low in cost.

[0063] Embodiments of the third aspect

[0064] The embodiment of the present application also provides a method for detecting the status of goods, which corresponds to the device for detecting the status of goods in the embodiment of the first aspect. Figure 7 FIG. 1 is a schematic diagram of a method for detecting the status of goods according to an embodiment of the present application. Figure 7 As shown, the method includes:

[0065] 701, extracting edge features from a first image to generate a first edge feature image, and extracting edge features from a second image to generate a second edge feature image;

[0066] 702, calculating a first difference image between the second edge feature image and the first edge feature image, and determining a product state change area according to the first difference image;

[0067] 703, determining the type of change in the product status in the product status change area.

[0068] In this embodiment, the specific implementation method of the above steps is the same as that described in the embodiment of the first aspect and will not be repeated here.

[0069] Figure 8 FIG. 1 is a schematic diagram of a method for detecting the status of goods according to an embodiment of the present application. Figure 8 As shown, the method includes:

[0070] 801, extracting edge features from a first image to generate a first edge feature image, and extracting edge features from a second image to generate a second edge feature image;

[0071] 802, calculating a first difference image between the second edge feature image and the first edge feature image, and determining a product state change area according to the first difference image; and executing 803 and 807;

[0072] 803, determining the type of change in the state of the goods in the state of the goods change region according to an edge classification algorithm, including: determining a first number of pixel points belonging to the edge feature of the second image in the state of the goods change region in the first difference image, and determining a second number of pixel points belonging to the edge feature of the first image, and determining the type of change in the state of the goods in the state of the goods change region according to a relationship between the first number, the second number, and a threshold;

[0073] 804, determine whether the type of change in the status of the goods is taken away or added, if it is determined to be yes, execute 806, otherwise execute 805;

[0074] 805, determining a type of change in the state of the goods in the state of the goods change area according to a depth difference algorithm, including: generating a first depth image according to the first image, generating a second depth image according to the second image, calculating a second difference image between the second depth image and the first depth image, and determining the type of change in the state of the goods in the state of the goods change area in the second difference image;

[0075] 806, output the type of change in the status of the goods;

[0076] 807, determining the type of change in the state of the goods in the state of the goods change region according to a neural network algorithm, including: using the same neural network to extract a first feature and a second feature from the state of the goods change region in the first image and the second image, calculating a feature difference between the first feature and the second feature, and determining the type of change in the state of the goods according to the feature difference;

[0077] 808, determining the final product status change type based on the product status change types determined in 806 and 807 (comparing confidence levels).

[0078] The implementation of 801 to 808 can refer to the embodiment of the first aspect and will not be described in detail here.

[0079] The above only describes the steps or processes related to the present application, but the present application is not limited to this. The product status detection method may also include other steps or processes. For the specific content of these steps or processes, reference may be made to the prior art. In addition, the above only uses some structures of the model used in the product status detection method as an example to illustrate the embodiments of the present application, but the present application is not limited to these structures, and these structures may be appropriately modified, and the implementation methods of these modifications should all be included in the scope of the embodiments of the present application.

[0080] The above embodiments are merely exemplary of the embodiments of the present application, but the present application is not limited thereto, and appropriate modifications may be made based on the above embodiments. For example, the above embodiments may be used alone, or one or more of the above embodiments may be combined.

[0081] It can be seen from the above embodiments that the area where the state of goods changes and the type of the state of goods change are determined based on the difference image of the edge images extracted from different images. Therefore, the automatic determination of the type of state of goods change can be achieved based only on the image acquired by the monocular camera, and the determination based on the difference image can improve the accuracy of the determination and is relatively low in cost.

[0082] An embodiment of the present application also provides a computer-readable program, wherein when the program is executed in a product status detection device or an electronic device, the program enables a computer to execute the product status detection method described in the third aspect embodiment in the product status detection device or the electronic device.

[0083] An embodiment of the present application also provides a storage medium storing a computer-readable program, wherein the computer-readable program enables a computer to execute the product status detection method described in the embodiment of the third aspect in a product status detection device or an electronic device.

[0084] The goods status detection device or the goods status detection method executed in the electronic device described in the embodiments of the present application can be directly embodied as hardware, a software module executed by a processor, or a combination of the two. For example, Figure 1 One or more of the functional block diagrams shown in and / or one or more combinations of the functional block diagrams may correspond to various software modules of the computer program flow or to various hardware modules. These software modules may correspond to Figures 7 and 8 These hardware modules can be implemented by solidifying these software modules using, for example, a field programmable gate array (FPGA).

[0085] The software module may be located in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. A storage medium may be coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium; or the storage medium may be an integral part of the processor. The processor and the storage medium may be located in an ASIC. The software module may be stored in a memory of a mobile terminal or in a memory card that can be inserted into the mobile terminal. For example, if the electronic device uses a large-capacity MEGA-SIM card or a large-capacity flash memory device, the software module may be stored in the MEGA-SIM card or the large-capacity flash memory device.

[0086] against Figure 1 One or more of the functional block diagrams and / or one or more combinations of the functional block diagrams described herein may be implemented as a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, or any appropriate combination thereof for performing the functions described in the present application. Figure 1 One or more of the described functional block diagrams and / or one or more combinations of the functional block diagrams may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in communication with a DSP, or any other such configuration.

[0087] The present application is described above in conjunction with specific implementation methods, but it should be clear to those skilled in the art that these descriptions are exemplary and are not intended to limit the scope of protection of the present application. Those skilled in the art can make various modifications and variations to the present application based on the spirit and principles of the present application, and these modifications and variations are also within the scope of the present application.

[0088] This application also provides the following notes:

[0089] 1. A method for detecting a product status, characterized in that the method comprises:

[0090] Extract edge features from the first image to generate a first edge feature image, and extract edge features from the second image to generate a second edge feature image;

[0091] Calculating a first difference image between the second edge feature image and the first edge feature image, and determining a product state change area according to the first difference image;

[0092] Determine the type of change in the status of the product in the product status change area.

[0093] 2. The method according to Note 1, wherein the edge feature is extracted using an edge detection operator, and the edge detection operator includes at least one of the following operators: Prewitt operator, Laplacian operator, Roberts crossover operator, Canny operator, Sobel operator.

[0094] 3. According to the method described in Note 1, determining the area where the state of the goods changes based on the first difference image also includes: performing noise reduction processing and graphics closing operations on the first difference image to generate the area where the state of the goods changes.

[0095] 4. The method according to Note 1, wherein determining the type of change in the state of the goods in the state change area of ​​the goods comprises:

[0096] A first depth image is generated according to the first image, a second depth image is generated according to the second image, a second difference image between the second depth image and the first depth image is calculated, and a type of change in the state of the goods in the area where the state of the goods changes is determined in the second difference image.

[0097] 5. The method described in Note 4 further comprises: standardizing the product change area in the second difference image, and determining the type of change in the product state according to the depth difference corresponding to the pixels in the product change area after the standardization.

[0098] 6. The method according to Note 5, wherein the type of change in the state of the goods is determined according to the depth difference corresponding to the pixels in the goods change area after standardization: the type of change in the state of the goods is determined according to the proportional relationship between the positive and negative values ​​of the depth difference in the goods change area after standardization, or binary clustering is performed on the depth difference in the goods change area after standardization, and the type of change in the state of the goods is determined according to the positive and negative relationship between the center points of different categories.

[0099] 7. The method according to Note 1 or 4, wherein determining the type of change in the state of the goods in the state of the goods change area comprises: determining a first number of pixel points belonging to edge features of the second image in the state of the goods change area in the first difference image, and determining a second number of pixel points belonging to edge features of the first image, and determining the type of change in the state of the goods in the state of the goods change area based on the relationship between the first number, the second number and a threshold.

[0100] 8. According to the method described in Note 7, determining the type of change in the status of the goods in the status of the goods area includes: first determining whether the type of change in the status of the goods in the status of the goods area is removal or addition, and when the type of change in the status of the goods is not removal or addition, the difference determination module determines the type of change in the status of the goods.

[0101] 9. The method according to Note 1, wherein determining the type of change in the state of the goods in the state of the goods change area includes: using the same neural network to extract a first feature and a second feature from the state of the goods change area in the first image and the second image, calculating a feature difference between the first feature and the second feature, and determining the type of change in the state of the goods based on the feature difference.

[0102] 10. A storage medium storing a computer-readable program, wherein the computer-readable program causes a processor coupled to the storage medium to execute the following method:

[0103] Extract edge features from the first image to generate a first edge feature image, and extract edge features from the second image to generate a second edge feature image;

[0104] Calculating a first difference image between the second edge feature image and the first edge feature image, and determining a product state change area according to the first difference image;

[0105] Determine the type of change in the status of the product in the product status change area.

Claims

1. A device for detecting the state of goods, characterized in that: The device comprises: A first detection unit, which extracts edge features from the first image to generate a first edge feature image, and extracts edge features from the second image to generate a second edge feature image; A first determining unit, which calculates a first difference image between the second edge feature image and the first edge feature image, and determines a product state change area according to the first difference image; The second determining unit determines a type of a change in the state of the product in the state of the product change area.

2. The device according to claim 1, wherein: The first detection unit extracts the edge feature using an edge detection operator, and the edge detection operator includes at least one of the following operators: Prewitt operator, Laplacian operator, Roberts crossover operator, Canny operator, Sobel operator.

3. The device according to claim 1, wherein: The first determining unit performs noise reduction processing and graphics closing operation on the first difference image to generate the product state change area.

4. The device according to claim 1, wherein: The second determination unit includes: a difference determination module; The difference determination module generates a first depth image based on the first image, generates a second depth image based on the second image, calculates a second difference image between the second depth image and the first depth image, and determines the type of change in the state of the goods in the area where the state of the goods changes in the second difference image.

5. The device according to claim 4, wherein: The difference determination module further performs standardization processing on the product change area in the second difference image, and determines the product state change type according to the depth difference corresponding to the pixels in the product change area after the standardization processing.

6. The device according to claim 5, wherein: The difference determination module determines the type of change in the state of the goods according to the proportional relationship between the positive and negative values ​​of the depth difference in the goods change area after standardization, or performs binary clustering on the depth difference in the goods change area after standardization, and determines the type of change in the state of the goods according to the positive and negative relationship of the center points of different categories.

7. The device according to claim 1 or 4, wherein: The second determination unit includes: an edge determination module; The edge determination module determines a first number of pixel points belonging to edge features of the second image in the product state change area in the first difference image, and determines a second number of pixel points belonging to edge features of the first image, and determines a type of product state change in the product state change area based on a relationship among the first number, the second number and a threshold.

8. The device according to claim 7, wherein: The edge determination module first determines whether the product status change type of the product status change area is removal or addition. When the product status change type is not removal or addition, the difference determination module determines the product status change type.

9. The device according to claim 1, wherein: The second determination unit includes: a neural network determination module; The neural network determination module uses the same neural network to extract the first feature and the second feature from the product state change area in the first image and the second image, calculates the feature difference between the first feature and the second feature, and determines the type of product state change based on the feature difference.

10. A method for detecting the status of goods, characterized in that: The method comprises: Extract edge features from the first image to generate a first edge feature image, and extract edge features from the second image to generate a second edge feature image; Calculating a first difference image between the second edge feature image and the first edge feature image, and determining a product state change area according to the first difference image; Determine the type of change in the status of the product in the product status change area.

Images