Product status detection device and product status detection method

Abstract

To provide a product status detection device and a product status detection method.SOLUTION: A production status detection device includes: a first detection unit which extracts edge features from a first image to generate a first edge feature image and extracts edge features from a second image to generate a second edge feature image; a first determination unit which calculates a first difference image between the second edge feature image and the first edge feature image and determines a product status change region on the basis of the first difference image; and a second determination unit which determines a product status change type in the product status change region.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to the technical field of image processing. [Background technology]

[0002] In the traditional retail industry, supermarkets and convenience stores require full-time staff, and labor costs are relatively high. In the related art, staff are required to check whether products on the shelves are replenished, out of stock, or moved by customers. At present, a commonly used change detection method is to find the changed parts in the images by comparing images taken at different times, and then make a judgment. Such a change detection method is usually applied to outdoor scenes, such as satellite remote sensing, because in the case of outdoor scenes, image pairs are taken by satellites, most of the contents in the image pairs have changed, and it is sufficient to detect the changed areas. However, when it comes to indoor retail scenes, the image pairs are taken by fixed surveillance camera heads, most of the contents in the image pairs have not changed, and there are a large number of products on the shelves that are dense and have similar textures and shapes, making it more difficult to implement the above change detection method.

[0003] It should be noted that the introduction of the above background art is intended to clearly and completely explain the technical solutions of the present invention, and to facilitate understanding thereof by those skilled in the art, and these technical solutions are described in the background art of the present invention, and therefore should not be construed as being known to those skilled in the art. Summary of the Invention [Problem to be solved by the invention]

[0004] In view of at least one of the above problems, embodiments of the present invention aim to provide an apparatus and method for detecting a product condition. [Means for solving the problem]

[0005] According to a first aspect of an embodiment of the present invention, there is provided an item condition detection device, the device comprising: a first detection unit that extracts edge features from the first image to generate a first edge feature image and extracts edge features from the second image to generate a second edge feature image; a first determination unit for calculating a first difference image between the second edge feature image and the first edge feature image, and determining a product state change area based on the first difference image; and The second determining unit is configured to determine a product state change type in the product state change area.

[0006] According to a second aspect of an embodiment of the present invention, there is provided a method for detecting a product condition, the method comprising: Extracting edge features from the first image to generate a first edge feature image, and extracting edge features from the second image to generate a second edge feature image; Calculating a first difference image between the second edge feature image and the first edge feature image, and determining a product state change area based on the first difference image; and The method includes determining a product status change type in the product status change area. Effect of the Invention

[0007] One of the advantageous effects of the embodiment of the present invention is as follows: the product condition change area and the product condition change type are determined based on the difference image of the edge images extracted from different images, thereby enabling automated determination of the product condition change type based only on the image captured by the monocular camera. Furthermore, by making the determination based on the difference image, the accuracy (precision) of the determination can be improved and the cost is relatively low.

[0008] It should be noted that terms such as "comprise / have" when used in this specification refer to the presence of a feature, element, step, or assembly, but do not exclude the presence or addition of one or more other features, elements, steps, or assemblies. [Brief description of the drawings]

[0009] Elements and features described in one drawing or one embodiment of the invention may be combined with elements and features shown in one or more other drawings or embodiments, and in the drawings, like reference numerals are used to indicate corresponding parts in several drawings and to indicate corresponding parts used in multiple embodiments. [Figure 1] FIG. 1 is a diagram showing a commodity state detection device according to an embodiment of the present invention. [Diagram 2] FIG. 11 is a diagram showing how a product state change area is determined based on a first differential image in an embodiment of the present invention. [Diagram 3] FIG. 2 shows a depth image and a second difference image in an embodiment of the present invention. [Figure 4] FIG. 2 is a diagram showing a neural network configuration in an embodiment of the present invention. [Diagram 5] 1 is a diagram showing an electronic device according to an embodiment of the present invention; [Figure 6] 1 is a block diagram showing a system configuration of an electronic device according to an embodiment of the present invention. [Figure 7] FIG. 2 is a diagram showing a method for detecting a commodity state in an embodiment of the present invention. [Figure 8] FIG. 2 is a diagram showing a method for detecting a commodity state in an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] The above and other features of the present invention will become apparent from the following detailed description and the accompanying drawings, in which: While the specification and drawings disclose certain embodiments of the present invention, these are illustrative of only some of the embodiments which may employ the principles of the present invention, and it is to be understood that the present invention is not limited to the described embodiments, but rather includes all modifications, variations and alternatives which fall within the scope of the appended claims.

[0011] <Example of the first aspect> An embodiment of the present invention provides an item condition detection device. Fig. 1 is a diagram showing an item condition detection device in an embodiment of the present invention. As shown in Fig. 1, an item condition detection device 100 includes:

[0012] A first detection unit 101: extracts edge features from a first image to generate a first edge feature image, and extracts edge features from a second image to generate a second edge feature image; A first determination unit 102: calculates a first difference image between the second edge feature image and the first edge feature image, and determines a product state change area according to the first difference image; and A second determination unit 103: determines the product state change type in the product state change area.

[0013] In some embodiments, the first and second images may be two frames of images in a video stream captured by the same imager, which may be obtained from an imager capturing an area where the product is located, for example, captured by a surveillance camera head installed, the imager being a monocular RGB camera, after capturing one frame of real-time surveillance video, the image of the one frame is taken as the first image, and after a predetermined time has passed, another frame of image is captured from the surveillance video, and the image of the other frame is taken as the second image, and the predetermined time may be one day, one hour, 12 hours, or one week, etc., but the embodiments of the present invention are not limited thereto. The first and second images may be two-dimensional color images (RGB) or two-dimensional grayscale images, but the embodiments of the present invention are not limited thereto, and the sizes of the first and second images are the same or are processed to be the same.

[0014] In some embodiments, the first detection unit 101 may use an edge detection operator to extract edge features from the first image to generate a first edge feature image, and extract edge features from the second image to generate a second edge feature image, where the edge detection operator may include at least one of the following operators, but is not limited to: Prewitt operator, Laplacian operator, Roberts cross operator, Canny operator, Sobel operator, etc. The first edge feature image and the second edge feature image may be binary images, that is, edge feature parts are displayed in white or gray and other parts are displayed in black, and the edge feature parts in the first edge feature image and the second edge feature image may be displayed in the same color or different colors, but the embodiments of the present invention are not limited thereto.

[0015] In some embodiments, the second image can be regarded as an image after the product state changes, and the first image can be regarded as an image before the product state changes, and the first determination unit 102 calculates the difference (difference in pixel values ​​at corresponding positions) between the second edge feature image and the first edge feature image to obtain a first difference image (e.g., a binary image), and determines a product state change area based on the first difference image, where, if there is no product state change area in the image, the pixels at corresponding positions in the second edge feature image and the first edge feature image are the same, the pixel difference is 0, and may be displayed as black in the first difference image; if there is a product state change area in the image, the pixels at corresponding positions in the second edge feature image and the first edge feature image are different, the pixel difference is not 0, and may be displayed as white and / or gray in the first difference image. In other words, the pixel area in the first difference image whose pixel value is not 0 can be the product state change area.

[0016] In some embodiments, after obtaining the first difference image, the first determination unit can further perform a noise reduction process and a graphics closing process on the first difference image to generate the product state change area. Figure 2 is a diagram showing the determination of the product state change area based on the first difference image in an embodiment of the present invention. As shown in Figure 2(a), the non-black (non-zero pixel value) pixel part in the first difference image is the product state change area. In order to avoid the influence of light and shadow, the image in b can be obtained by performing a noise reduction process on the first difference image. The noise reduction process includes a process of removing pixel areas in the first difference image where the area of ​​connected pixels (area of ​​the area consisting of adjacent pixels) is smaller than a predetermined area, and the removing operation can be regarded as setting the pixel value of the corresponding pixel area to 0. The image after noise reduction is subjected to a graphics closing process to obtain an image at c, where the non-black pixel portion (pixel value is not 0) at c is the product state change area, and the principle of the closing process is to first perform expansion and then erosion, thus removing the void inside the connection area, filling the void by performing expansion, and then performing erosion to restore the connection area, thereby obtaining a better product state change area. Note that the closing process can be referred to in the related art, and will not be described in detail in the embodiments of the present invention.

[0017] As can be seen from the above embodiments, the determined product condition change area can be represented by the coordinates of pixel points whose pixel values ​​are not 0 in the first difference image or the first difference image after noise reduction processing and closing processing, or a minimum boundary frame surrounding pixel points whose pixel values ​​are not 0 in the first difference image or the first difference image after noise reduction processing and closing processing can be determined, and the product condition change area can be represented by the coordinates of the minimum boundary frame, but the embodiments of the present invention are not limited to these.

[0018] In some embodiments, after determining the product status change area, the second determination unit 103 can further determine a product status change type in the product status change area, the product status change type including takeaway (out of stock), add (replenish) and move, the embodiment of the present invention proposes to use three algorithms, namely, a depth difference algorithm, an edge classification algorithm and a neural network algorithm, to determine the product status change type in the product status change area, the three algorithms may be implemented independently or in combination, but the embodiment of the present invention is not limited thereto. The following will be described in detail.

[0019] In some embodiments, for the depth difference algorithm, the second determination unit 103 includes a difference determination module, which generates a first depth image according to the first image, generates a second depth image according to the second image, calculates a second difference image between the second depth image and the first depth image, and determines the product state change type of the product state change area in the second difference image. Since the first image and the second image are images acquired by a monocular RGB camera, the first depth image and the second depth image (which are grayscale images, and different grayscale values ​​represent different depths) can be acquired by using a conventional monocular depth estimation model, such as AdaBins, etc., that is, the first image is input into the monocular depth estimation model to obtain the first depth image, and the second image is input into the monocular depth estimation model to obtain the second depth image. The implementation manner of the monocular depth estimation model can be referred to in the related art, and the detailed description thereof will be omitted in the embodiments of the present invention. The difference determination module calculates the difference between the second depth image and the first depth image (the difference between pixel values ​​at corresponding positions), obtains a second difference image (grayscale image), finds the product state change area determined by the first determination unit 102 (the same pixel point position or the aforementioned minimum bounding box position) from the second difference image, and determines the product state change type within the product state change area.

[0020] In some embodiments, the difference determination module further performs a standardization process on the product change area in the second difference image, and determines the product state change type based on the depth difference corresponding to the pixel of the product change area after the standardization process. The standardization process includes normalizing the depth difference corresponding to the pixel point in the product change area within a predetermined range, for example, normalizing the depth difference corresponding to the pixel point in the product change area within a range of -1 to 1 according to a ratio. In other words, after the standardization process, 0 represents that the corresponding depth difference is unchanged, a positive value represents that the depth value is large, which corresponds to the fact that there was originally a product, but the product is taken away (carried away) after the change, so that the distance between the position and the camera becomes farther (larger), and a negative value represents that the depth value is small, which corresponds to the fact that there was originally no product, but the product is added (replenished) after the change, so that the distance between the position and the camera becomes closer (smaller). Note that the embodiment of the present invention is not limited to this. Figure 3 is a diagram showing a depth image and a second difference image in an embodiment of the present invention, where a in Figure 3 is the first depth image and the second depth image, b in Figure 3 is the second difference image, the position of the circle in the second difference image is the product state change area where the first determination unit is determined, and c in Figure 3 is a standardization processing image in which standardization processing has been performed on the product state change area.

[0021] In some embodiments, the difference determination module determines the product state change type based on the ratio of positive and negative depth differences in the product change area after standardization. For example, in the product change area after standardization, the number N1 of depth differences greater than 0 and the number N2 of depth differences smaller than 0 are counted (statistics) among the depth differences (after normalization) corresponding to each pixel, and the product state change type is determined based on the ratio of N1 and N2. For example, when N1 / (N1+N2) is greater than a predetermined value (e.g., 70%), the product state change type is represented as takeaway, when N2 / (N1+N2) is greater than a predetermined value (e.g., 70%), the product state change type is represented as addition, and otherwise, the product state change type is represented as movement. The predetermined value may be determined according to needs, and is not limited in the embodiments of the present invention.

[0022] In some embodiments, the difference determination module performs binary clustering on the depth difference in the product change area after standardization, and determines the product state change type according to the positive / negative relationship of the center points of different classes / categories. For example, the difference determination module may use a clustering algorithm to perform binary clustering on the depth difference in the product change area after standardization, including but not limited to the K-means algorithm, and after binary clustering, the depth difference in the product change area after standardization can be divided into two types, and the center points of different categories (for example, the average value of the depth difference in a category) are calculated. If the center points of the two categories are all positive, the product state change type is takeaway; if the center points of the two categories are all negative, the product state change type is addition; if one of the center points of the two categories is positive and the other is negative, the product state change type is movement.

[0023] In some embodiments, for the edge classification algorithm, the second determination unit 103 includes an edge determination module, which determines a first number of pixel points belonging to the edge features of the second image in the product state change area in the first difference image, and determines a second number of pixel points belonging to the edge features of the first image. In the first difference image, in the product state change area, there are pixel points belonging to the edge features of the first image, and there are also pixel points belonging to the edge features of the second image, for example, when the edge feature parts in the second edge feature image and the first edge feature image are respectively represented by white and gray, in the product state change area of ​​the first difference image, the number of pixel points of gray lines is the second number, and the number of pixel points of white lines is the first number.

[0024] In some embodiments, the edge determination module determines the product status change type of the product status change area according to the relationship between the first quantity, the second quantity, and a threshold value, for example, when the first quantity / (first quantity+second quantity) is greater than the threshold value, it indicates that the first quantity is predominant, in which case the product status change type is add, when the second quantity / (first quantity+second quantity) is greater than the threshold value, it indicates that the second quantity is predominant, in which case the product status change type is takeaway, otherwise the product status change type is move. The threshold value may be determined according to needs, for example, it may be set to 0.6, but the embodiment of the present invention is not limited thereto.

[0025] In some embodiments, the depth difference algorithm and the edge classification algorithm may be implemented alone or in combination. For example, when implemented in combination (hereinafter referred to as a hybrid algorithm), the edge determination module first determines whether the product status change type of the product status change area is takeaway or add, and then, when the product status change type is neither takeaway nor add, the difference determination module determines the product status change type. This can further improve the detection accuracy (precision) of the product status change type.

[0026] In some embodiments, for the neural network algorithm, the second determination unit 103 includes a neural network determination module, which uses the same neural network to extract first and second features from the product state change areas in the first and second images, calculate a feature difference between the first and second features, and determine the product state change type based on the feature difference. Figure 4 is a diagram showing the neural network configuration in an embodiment of the present invention. As shown in FIG. 4 , after the first determination unit determines the product state change area, a first change area image I1 and a second change area image I2 are extracted from the original first image and second image according to the position of the product state change area. The first change area image and the second change area image are input into the same convolutional neural network (where the shared weight, structure, and parameters are all the same). The outputs of the two convolutional neural networks are the first feature and the second feature, respectively. The feature difference between the first feature and the second feature is calculated. The feature difference is input into a fully connected network. The output of the fully connected network is the product state change type. The convolutional neural network and the fully connected network can be trained using pre-collected image pairs and labeled product state change types as training data, but the embodiment of the present invention is not limited thereto.

[0027] In some embodiments, the hybrid algorithm and the neural network algorithm may be implemented alone or in combination. When implemented in combination, the product condition change type may be determined based on the hybrid algorithm and the neural network algorithm, respectively, and a comprehensive judgment may be made based on the determination results of both, for example, a first reliability of the product condition change type determined by the hybrid algorithm and a second reliability of the product condition change type determined by the neural network algorithm are respectively determined, and if the first reliability is greater than the second reliability, the product condition change type is determined to be the product condition change type determined by the hybrid algorithm, and if the first reliability is less than or equal to the second reliability, the product condition change type is determined to be the product condition change type determined by the neural network algorithm. Wherein, the first reliability may be determined based on a ratio value calculated by a depth differencing algorithm or an edge classification algorithm (e.g., the calculated ratio value is the first reliability, or the first reliability corresponding to the ratio value corresponding to the different types is pre-determined based on a positive correlation between the ratio value and the first reliability), and the second reliability may be determined based on the probability of different types output from a neural network model (e.g., the probability of each type output from the neural network model is the second reliability).

[0028] Although only the components or modules related to the present invention have been described above, the present invention is not limited thereto. The product state detection device 100 may further include other components or modules, and the specific contents of these components or modules can be found in the related art.

[0029] 1 shows only the connection relationships or signal directions between each component or module, but as can be understood by those skilled in the art, various related technologies such as bus connection may be adopted. Also, each of the above-mentioned components or modules may be realized by hardware such as a processor, a memory, etc., but the embodiment of the present invention is not limited thereto.

[0030] The above-mentioned embodiments are merely for illustrative purposes, and the present invention is not limited thereto, and appropriate modifications can be made based on the above-mentioned embodiments. For example, the above-mentioned embodiments may be used alone, or a combination of the above-mentioned embodiments may be used.

[0031] As can be seen from the above embodiments, the product condition change area and the product condition change type are determined based on the difference image of the edge images extracted from different images, thereby enabling automated determination of the product condition change type based only on the image captured by the monocular camera. Furthermore, by making the determination based on the difference image, the accuracy of the determination can be improved and the cost is relatively low.

[0032] <Example of the second aspect> An electronic device is further provided in the embodiment of the present invention, and Fig. 5 is a diagram showing an electronic device in the embodiment of the present invention. As shown in Fig. 5, an electronic device 500 includes an item condition detection device 501, and the configuration and function of the item condition detection device 501 are the same as the item condition detection device 100 in the embodiment of the first aspect, and the specific contents can be referred to the description in the embodiment of the first aspect, and the detailed description thereof will be omitted here.

[0033] Fig. 6 is a block diagram showing a system configuration of an electronic device according to an embodiment of the present invention. As shown in Fig. 6, an electronic device 600 may include a processor 601 and a memory 602, and the memory 602 is connected to the processor 601. It should be noted that this figure is merely an example, and other types of components may be used to supplement or replace the components to realize telecommunication functions or other functions.

[0034] In some embodiments, the processor 601 includes at least one of a central processing unit (CPU) and a graphics processing unit (GPU).

[0035] As shown in FIG. 6, the electronic device 600 may further include an input device 603, a display device 604, a power source 605, etc. In one embodiment, the function of the product condition detection device can be integrated into the processor 601. The processor 601 may be configured to: extract edge features from a first image to generate a first edge feature image, extract edge features from a second image to generate a second edge feature image; calculate a first difference image between the second edge feature image and the first edge feature image, and determine a product condition change area based on the first difference image; and determine a product condition change type in the product condition change area. Note that, for other embodiments, reference may be made to the embodiments of the first aspect, and detailed descriptions thereof will be omitted here.

[0036] In another implementation manner, the product condition detection device may be located separately from the processor 601, for example, the product condition detection device may be configured as a chip connected to the processor 601, and the function of the product condition detection device may be realized under the control of the processor 601.

[0037] In this embodiment, electronic device 600 need not include all of the components shown in FIG.

[0038] As shown in FIG. 6 , the processor 601, which may be referred to as a controller or operational control, may include a microprocessor or other processing and / or logic device, and the processor 601 can receive inputs and control the operation of each component of the electronic device 600.

[0039] The storage 602 may be, for example, one or more of a buffer, a flash memory, a HDD, a removable medium, a volatile storage, a non-volatile storage, or other suitable devices, and the processor 601 can execute the program stored in the storage 602 to realize information storage or processing. The functions of the other components are the same as those of the conventional components, and therefore detailed descriptions thereof are omitted here. Each component of the electronic device 600 may be realized by dedicated hardware, firmware, software, or a combination thereof, all of which are within the scope of the present invention.

[0040] As can be seen from the above embodiments, the product condition change area and the product condition change type are determined based on the difference image of the edge images extracted from different images, thereby realizing automated judgment of the current product condition change type based only on the image captured by the monocular camera. Furthermore, by making the judgment based on the difference image, the accuracy of the judgment can be improved and the cost is relatively low.

[0041] <Example of the third aspect> An embodiment of the present invention further provides a merchandise condition detection method, which corresponds to the merchandise condition detection device in the embodiment of the first aspect. Figure 7 is a diagram showing the merchandise condition detection method in the embodiment of the present invention. As shown in Figure 7, the method includes the following steps:

[0042] 701: Extract edge features from a first image to generate a first edge feature image, and extract edge features from a second image to generate a second edge feature image; 702: Calculating a first difference image between the second edge feature image and the first edge feature image, and determining a product state change area based on the first difference image; and 703: A product status change type in the product status change area is determined.

[0043] In this embodiment, the specific implementation methods of the above steps are the same as those described in the embodiment of the first aspect, and detailed descriptions thereof will be omitted here.

[0044] 8 is a diagram showing a method for detecting a commodity state in an embodiment of the present invention. As shown in FIG. 8, the method includes the following steps:

[0045] 801: Extracting edge features from a first image to generate a first edge feature image, and extracting edge features from a second image to generate a second edge feature image; 802: Calculate a first difference image between the second edge feature image and the first edge feature image, and determine a product state change area based on the first difference image, and then execute 803 and 807; 803: Determining a product state change type of the product state change area by an edge classification algorithm, which includes: determining a first quantity of pixel points belonging to edge features of the second image in the product state change area in the first difference image, and determining a second quantity of pixel points belonging to edge features of the first image; and determining a product state change type of the product state change area according to a relationship between the first quantity, the second quantity and a threshold value; 804: Determine whether the product status change type is take-out or add-on. If it is determined as Yes, execute 806. If it is determined as No, execute 805; 805: Determine a product state change type of the product state change area by a depth difference algorithm, which includes generating a first depth image according to the first image, generating a second depth image according to the second image, calculating a second difference image between the second depth image and the first depth image, and determining a product state change type of the product state change area in the second difference image; 806: Output product status change type; 807: determining a product state change type of the product state change area by a neural network algorithm, which includes using the same neural network to extract a first feature and a second feature from the product state change area in the first image and the second image, calculating a feature difference between the first feature and the second feature, and determining the product state change type based on the feature difference; and 808: A final product condition change type is determined based on the product condition change types determined by 806 and 807 (comparing the reliability).

[0046] The implementation of 801 to 808 can be referred to in the embodiment of the first aspect, and detailed description thereof will be omitted here.

[0047] The above describes only the steps or processes related to the present invention, but the present invention is not limited thereto. The product condition detection method may further include other steps or processes, and the specific contents of these steps or processes can be referred to the prior art. In addition, the above describes the embodiments of the present invention by taking several configurations of the model used in the product condition detection method as examples, but the present invention is not limited to these configurations, and furthermore, appropriate modifications can be made to these configurations, and all of the implementation methods of these modifications fall within the scope of the embodiments of the present invention.

[0048] The above-mentioned embodiments are merely for illustrative purposes, and the present invention is not limited thereto. Furthermore, appropriate modifications may be made based on the above-mentioned embodiments. For example, the above-mentioned embodiments may be used alone, or a combination of two or more of the above-mentioned embodiments may be used.

[0049] As can be seen from the above embodiments, the product condition change area and the product condition change type are determined based on the difference image of the edge images extracted from different images, thereby enabling automated determination of the product condition change type based only on the image captured by the monocular camera. Furthermore, by making the determination based on the difference image, the accuracy of the determination can be improved and the cost is relatively low.

[0050] In a further embodiment of the present invention, a computer-readable program is provided, in which, when the program is executed on a product condition detection device or electronic device, the program causes a computer to execute the product condition detection method described in the embodiment of the third aspect on the product condition detection device or electronic device.

[0051] In a further embodiment of the present invention, a storage medium storing a computer readable program is provided, in which the computer readable program causes a computer to execute the product condition detection method described in the embodiment of the third aspect in a product condition detection device or electronic device.

[0052] The above-mentioned devices and methods may be realized by software or hardware, or may be realized by a combination of hardware and software. The present invention further relates to a computer-readable program, which, when executed by a logic component, causes the logic component to realize the above-mentioned devices or components, or causes the logic component to realize the above-mentioned methods or steps. The logic component may be, for example, a Field Programmable Gate Array (FPGA), a microprocessor, a processor for use in a computer, etc. The present invention also relates to a storage medium, such as a hard disk, a magnetic disk, an optical hard disk, a DVD, a flash memory, etc., storing the above-mentioned program.

[0053] Additionally, one or more combinations of the functional blocks depicted in the figures and / or one or more combinations of the functional blocks may be implemented as a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic component, a discrete gate or transistor logic component, a discrete hardware assembly, or any other suitable combination for performing the functions described herein. Additionally, one or more combinations of the functional blocks depicted in the figures and / or one or more combinations of the functional blocks may further be configured as a combination of computing devices, such as a combination of a DSP and a microprocessor, a combination of multiple microprocessors, one or more microprocessors in communication with a DSP, or any other configuration.

[0054] Furthermore, with respect to the above-mentioned embodiments, the following supplementary notes are disclosed.

[0055] (Appendix 1) A commodity condition detection method, comprising: Extracting edge features from the first image to generate a first edge feature image, and extracting edge features from the second image to generate a second edge feature image; Calculating a first difference image between the second edge feature image and the first edge feature image, and determining a product state change area based on the first difference image; and determining a product condition change type in the product condition change area.

[0056] (Appendix 2) 2. The method according to claim 1, comprising: A method comprising: extracting the edge features using an edge detection operator, the edge detection operator including at least one of the following operators: a Prewitt operator, a Laplacian operator, a Roberts intersection operator, a Canny operator, and a Sobel operator.

[0057] (Appendix 3) 2. The method according to claim 1, comprising: The method, wherein determining a product state change area based on the first difference image further includes performing a noise reduction process and a graphics closing process on the first difference image to generate the product state change area.

[0058] (Appendix 4) 2. The method according to claim 1, comprising: Determining the type of change in the product state in the product state change area includes: The method includes generating a first depth image based on the first image, generating a second depth image based on the second image, calculating a second difference image between the second depth image and the first depth image, and determining a product state change type of the product state change area in the second difference image.

[0059] (Appendix 5) The method according to claim 4, further comprising: The method includes performing a standardization process on the product change area in the second difference image, and determining the product state change type based on the depth difference corresponding to the pixels of the product change area after the standardization process.

[0060] (Appendix 6) 6. The method according to claim 5, The method includes determining the product state change type based on a depth difference corresponding to pixels in the product change area after standardization processing, determining the product state change type based on a relationship between the ratio of positive and negative values ​​of the depth difference in the product change area after standardization processing, or performing binary clustering on the depth difference in the product change area after standardization processing, and determining the product state change type based on the positive and negative relationship of center points of different categories.

[0061] (Appendix 7) 5. The method according to claim 1 or 4, Determining a product condition change type in the product condition change area includes determining a first quantity of pixel points belonging to edge features of the second image and determining a second quantity of pixel points belonging to edge features of the first image in the product condition change area in the first differential image, and determining a product condition change type of the product condition change area based on a relationship between the first quantity, the second quantity and a threshold value.

[0062] (Appendix 8) 8. The method according to claim 7, determining a product status change type in the product status change area includes first determining whether the product status change type in the product status change area is take-out or add-on, and then, when the product status change type is not take-out or add-on, the difference determination module determines the product status change type.

[0063] (Appendix 9) 2. The method according to claim 1, comprising: determining a product condition change type in the product condition change area includes: extracting a first feature and a second feature from the product condition change area in the first image and the second image using the same neural network, calculating a feature difference between the first feature and the second feature, and determining the product condition change type based on the feature difference.

[0064] (Appendix 10) A storage medium storing a computer-readable program, The computer readable program causes a processor coupled to the storage medium to execute the following method: Extracting edge features from the first image to generate a first edge feature image, and extracting edge features from the second image to generate a second edge feature image; Calculating a first difference image between the second edge feature image and the first edge feature image, and determining a product state change area based on the first difference image; and determining a product status change type in the product status change area.

[0065] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to such embodiments, and any modifications to the present invention fall within the technical scope of the present invention as long as they do not depart from the spirit of the present invention.

Claims

1. A device for detecting the state of a commodity, a first detection unit for extracting edge features from the first image to generate a first edge feature image and for extracting edge features from the second image to generate a second edge feature image; a first determination unit for calculating a first difference image between the second edge feature image and the first edge feature image, and determining a product state change area based on the first difference image; and The apparatus further comprises a second determining unit for determining a product status change type in the product status change area.

2. 2. The apparatus of claim 1, The apparatus, wherein the first detection unit extracts the edge features using an edge detection operator, the edge detection operator including at least one of a Prewitt operator, a Laplacian operator, a Roberts intersection operator, a Canny operator, and a Sobel operator.

3. 2. The apparatus of claim 1, The first determination unit performs noise reduction processing and graphics closing processing on the first difference image to generate the product state change area.

4. 2. The apparatus of claim 1, The second determination unit includes a difference determination module; The difference determination module generates a first depth image based on the first image, generates a second depth image based on the second image, calculates a second difference image between the second depth image and the first depth image, and determines a product state change type of the product state change area in the second difference image.

5. 5. The apparatus of claim 4, The difference determination module further performs a standardization process on the product change area in the second difference image, and determines the product state change type based on the depth difference corresponding to the pixels of the product change area after the standardization process.

6. 6. The apparatus of claim 5, The difference determination module determines the product state change type based on the ratio of positive and negative values ​​of depth differences in the product change area after standardization processing, or performs binary clustering on the depth differences in the product change area after standardization processing, and determines the product state change type based on the positive and negative center points of different classes.

7. 5. The apparatus of claim 4, The second determination unit includes an edge determination module; The edge determination module determines a first quantity of pixel points belonging to edge features of the second image in the product condition change area in the first difference image, and determines a second quantity of pixel points belonging to the edge features of the first image, and determines a product condition change type of the product condition change area based on the relationship between the first quantity, the second quantity and a threshold value.

8. 8. The apparatus of claim 7, First, the edge determination module determines whether a product status change type of the product status change area is out of stock or replenishment, and then, when the product status change type is not out of stock or replenishment, the difference determination module determines the product status change type.

9. 2. The apparatus of claim 1, The second determination unit includes a neural network determination module; the neural network determination module uses the same neural network to extract first and second features from the product state change areas in the first and second images, calculate a feature difference between the first and second features, and determine the product state change type based on the feature difference.

10. A method for detecting a condition of an item, comprising: Extracting edge features from the first image to generate a first edge feature image, and extracting edge features from the second image to generate a second edge feature image; Calculating a first difference image between the second edge feature image and the first edge feature image, and determining a product state change area based on the first difference image; and determining a product condition change type in the product condition change area.

Images