A method for locating an item, a method for identifying an item, and an apparatus for doing so.

Abstract

This application belongs to the field of image processing technology and discloses an object localization method, an object recognition method, and an apparatus. The method includes: acquiring a real-time image sequence of an object detection area using an image acquisition device; obtaining a dynamic background image based on the real-time image sequence; acquiring a difference image between every two adjacent image frames in the real-time image sequence, and obtaining an average difference image based on each difference image; acquiring the sum of pixel values ​​in the difference image between the current image frame and the previous image frame in the real-time image sequence, and calculating the current difference image between the current image frame and the dynamic background image when the sum of pixel values ​​is less than a preset threshold; performing dot product and inversion calculations on the current difference image based on the average difference image to obtain an object localization image, and determining the location of the object region as the connected component with the largest area in the object localization image. This application can improve the accuracy of object localization.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an object positioning method, an object recognition method and apparatus. Background Technology

[0002] Image recognition, as a mainstream practical application of deep learning algorithms, has long played a role in various aspects of life, such as product recognition in unmanned shelves and smart retail cabinets. In scenarios like unmanned shelves and smart retail cabinets, the position of items in an image often needs to be located before product recognition. Existing item localization methods mainly fall into two categories: one uses traditional localization algorithms such as frame difference or contour detection, and the other uses deep learning object detection algorithms. However, existing frame difference methods require pre-acquiring background images and a stable shooting environment; once the shooting environment changes, the localization accuracy deteriorates. Contour detection methods have high requirements for the image background; the background must be clean and not similar in color to the product. When the background and product colors are similar, it is difficult to accurately locate the item. Deep learning object detection algorithms require pre-collecting a large number of item images for feature annotation, and the computation is time-consuming. For new products, the model needs to be retrained, resulting in poor adaptability. It is evident that in existing technologies, traditional item localization methods suffer from the problem of localization results being easily affected by changes in the external environment, while deep learning object detection algorithms suffer from long computation times and poor versatility. Summary of the Invention

[0003] This application provides an item positioning method, an item identification method, and an apparatus, which can improve the accuracy of item positioning.

[0004] In a first aspect, embodiments of this application provide a method for locating an item, the method comprising:

[0005] The real-time image sequence of the object detection area is acquired by an image acquisition device, and the real-time image sequence includes multiple consecutive image frames.

[0006] A dynamic background image is obtained based on the real-time image sequence;

[0007] Obtain the difference map between every two adjacent image frames in the real-time image sequence, and obtain the average difference map based on each difference map;

[0008] Obtain the sum of pixel values ​​in the difference map between the current image frame and the previous image frame in the real-time image sequence, and calculate the current difference map between the current image frame and the dynamic background map when the sum of pixel values ​​is less than a preset threshold.

[0009] The item location map is obtained by performing dot product and inversion on the current difference map based on the average difference map, and the connected region with the largest area in the item location map is determined as the location of the item region.

[0010] In one embodiment, a dynamic background image is obtained based on a real-time image sequence, including:

[0011] The dynamic background image is obtained by calculating the average value of all image frames in the real-time image sequence.

[0012] In one embodiment, a difference map is obtained between every two adjacent image frames in a real-time image sequence, and an average difference map is obtained based on each difference map, including:

[0013] Preprocessing is performed on every two adjacent image frames in the real-time image sequence to obtain multiple pairs of preprocessed adjacent image frames;

[0014] The difference map of each pair of preprocessed adjacent image frames is calculated using a structural similarity algorithm, and the average difference map is obtained based on each difference map.

[0015] In one embodiment, preprocessing is performed on every two adjacent image frames in the real-time image sequence to obtain multiple pairs of preprocessed adjacent image frames, including:

[0016] Each image frame in every two adjacent image frames is scaled to obtain multiple pairs of scaled adjacent image frames.

[0017] The difference map of each pair of preprocessed adjacent image frames is calculated using a structural similarity algorithm, and the average difference map is obtained based on each difference map, including:

[0018] The difference map of each pair of scaled adjacent image frames is calculated using a structural similarity algorithm, and the average difference map is obtained based on each difference map. The size of the average difference map is the same as the scaled size.

[0019] Based on the average difference map, the current difference map is multiplied and inverted to obtain the item location map, including:

[0020] The average difference map is restored to the size of the current image frame and used as the weight map;

[0021] The item location map is obtained by performing a dot product and inversion between the weighted map and the current difference map.

[0022] In one embodiment, a difference map for each pair of preprocessed adjacent image frames is calculated using a structural similarity algorithm, including:

[0023] Calculate the mean matrix and variance matrix of each image frame in each pair of preprocessed adjacent image frames, and calculate the covariance matrix between each pair of preprocessed adjacent image frames.

[0024] Based on the mean and variance matrices of each image frame in each pair of preprocessed adjacent image frames, and the covariance matrix between each pair of preprocessed adjacent image frames, a difference map of each pair of preprocessed adjacent image frames is obtained based on the structural similarity calculation formula.

[0025] In one embodiment, calculating the current difference map between the current image frame and the dynamic background map includes:

[0026] The current difference map between the current image frame and the dynamic background map is calculated using a structural similarity algorithm.

[0027] In one embodiment, the location of the item region is determined by the connected region with the largest area in the item location map, including:

[0028] Threshold segmentation and connected component extraction are performed on the item location map to determine each connected component in the item location map, and the connected component with the largest area among all connected components is determined as the location of the item region.

[0029] In one embodiment, a real-time image sequence of the object detection area is acquired via an image acquisition device, including:

[0030] In response to an image sequence acquisition command, detect the current state of the image acquisition device;

[0031] When the image acquisition device is in shooting mode, the current image frame captured by the image acquisition device and a preset number of consecutive image frames before the current image frame are acquired as a real-time image sequence.

[0032] When the image acquisition device is in the off state, control the image acquisition device to turn on and acquire multiple consecutive image frames captured by the image acquisition device in real time as a real-time image sequence.

[0033] The real-time image sequence is continuously updated as the image acquisition device continues to capture images.

[0034] In one embodiment, the method further includes:

[0035] When the sum of pixel values ​​in the difference map between the current image frame and the previous image frame in the real-time image sequence is less than a preset threshold, the acquisition of new image frames is stopped.

[0036] Secondly, this application provides an item identification method, which includes the item positioning method as described in any of the above embodiments, wherein the item region location is the region where the item to be identified is located;

[0037] The method also includes: using a trained deep learning neural network to identify the image of the location of the object region to obtain the identification result of the object to be identified.

[0038] Thirdly, embodiments of this application provide an article positioning device, which includes:

[0039] The image sequence acquisition module is used to acquire a real-time image sequence of the object detection area through an image acquisition device. The real-time image sequence includes multiple consecutive image frames.

[0040] The dynamic background image calculation module is used to obtain dynamic background images based on real-time image sequences.

[0041] The average difference map calculation module is used to obtain the difference map between every two adjacent image frames in a real-time image sequence, and to obtain the average difference map based on each difference map.

[0042] The current difference map calculation module is used to obtain the sum of pixel values ​​in the difference map between the current image frame and the previous image frame in the real-time image sequence, and to calculate the current difference map between the current image frame and the dynamic background map when the sum of pixel values ​​is less than a preset threshold.

[0043] The item location module is used to perform dot product and inversion calculations on the current difference map based on the average difference map to obtain the item location map, and to determine the location of the item region by the connected component with the largest area in the item location map.

[0044] Fourthly, this application provides an item recognition device, which includes an item recognition module and an item positioning device as described in any of the above embodiments, wherein the item area location is the area where the item to be identified is located.

[0045] The object recognition module is used to identify the location of an object in an image by using a trained deep learning neural network, and to obtain the recognition result of the object to be identified.

[0046] Fifthly, embodiments of this application provide an intelligent vending machine, including a processor, a memory, and an image acquisition device;

[0047] The image acquisition device is used to acquire real-time image sequences of the object detection area;

[0048] The memory is used to store computer programs. When the smart vending machine is running, the processor executes the computer programs to implement the steps of the item positioning method or item identification method as described in any of the above embodiments.

[0049] Sixthly, embodiments of this application provide an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the article positioning method or article identification method as described in any of the above embodiments.

[0050] In a seventh aspect, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the article positioning method or article identification method as described in any of the above embodiments.

[0051] In summary, compared with the prior art, the beneficial effects of the technical solution provided in this application include at least the following:

[0052] This application provides an item localization method, item recognition method, and apparatus. The item localization method can acquire a real-time image sequence of the item detection area through an image acquisition device. The real-time image sequence includes multiple consecutive image frames. Based on the real-time image sequence, real-time localization of items in the image is achieved. A dynamic background image is obtained based on the real-time image sequence. A difference image between every two adjacent image frames in the real-time image sequence is obtained, and an average difference image is obtained based on each difference image. Both the dynamic background image and the average difference image are dynamically updated as the real-time image sequence is updated. When the sum of pixel values ​​in the difference image between the current image frame and the previous image frame is less than a preset threshold, a current difference image between the current image frame and the dynamic background image is calculated. The current difference image is then multiplied and inverted based on the average difference image to obtain an item localization image. The connected component with the largest area in the item localization image is determined as the location of the item region. The above method does not require a pre-set background image, but dynamically updates the background image during the localization process, avoiding interference from changes in the external environment. At the same time, by using the average difference map as a weight map and performing dot product and inversion calculations with the current difference map, the interference caused by the object's movement on the background can be further reduced, thereby improving the accuracy of localization. In addition, compared with object detection algorithms, the above method can perform detection without relying on product features, has better versatility, can still complete localization on new items, and reduces the consumption of computing resources. Attached Figure Description

[0053] Figure 1 A flowchart of an article positioning method provided for an exemplary embodiment of this application.

[0054] Figure 2 A flowchart of the steps for obtaining the average difference map provided in an exemplary embodiment of this application.

[0055] Figure 3 A flowchart of an article identification method provided for another exemplary embodiment of this application.

[0056] Figure 4 This is a structural diagram of an article positioning device provided for an exemplary embodiment of this application.

[0057] Figure 5 A structural diagram of an article recognition device provided for another exemplary embodiment of this application. Detailed Implementation

[0058] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0059] Please see Figure 1 This application provides a method for locating an item, taking a terminal as the executing entity as an example. The method may include the following steps:

[0060] Step S1: Acquire a real-time image sequence of the object detection area using an image acquisition device. The real-time image sequence includes multiple consecutive image frames.

[0061] The image acquisition device can be any type of device, such as a camera, a document scanner, or a high-speed continuous shooting camera. In practical applications such as unmanned shelves and smart retail cabinets, the aforementioned image acquisition device is usually installed on the main equipment of these devices to capture real-time video images of the item detection area, or to continuously capture multiple consecutive image frames of the item detection area at specified time intervals. The item detection area is generally the item pick-up / placement area of ​​the main equipment. Customers can place items in the pick-up / placement area and wait for the main equipment to recognize the item information before putting the item in or taking it out. Examples include the pick-up / placement area of ​​an unmanned shelf and the item retrieval area of ​​a smart retail cabinet.

[0062] In the specific implementation process, when it is necessary to locate objects in an image before image recognition, an image acquisition device is needed to collect image data of the object detection area. Therefore, it is necessary to first confirm whether the image acquisition device is currently in working condition, i.e., whether it is taking pictures. Step S1 may include the following steps:

[0063] In response to an image sequence acquisition command, the current state of the image acquisition device is detected.

[0064] The image sequence acquisition instruction can be a user-issued instruction to the terminal or an instruction built into the terminal. In practical application scenarios, unmanned shelves and smart retail cabinets are often equipped with gravity sensors, motion sensors, etc., to detect whether materials or goods are being taken or placed. In this case, the item recognition instruction can be an image sequence acquisition instruction that is automatically generated when the terminal detects that materials or goods are being taken or placed through the sensors. This allows the image acquisition device to acquire images that may contain items, eliminating the need to continuously acquire and process image frames, thus avoiding unnecessary data processing.

[0065] When the image acquisition device is in shooting mode, the current image frame captured by the image acquisition device and a number of consecutive image frames before the current image frame are acquired as a real-time image sequence. At this time, the acquired image frames include the current image frame and image frames within a certain period of time in the past, which can avoid missing previous valid image frames and help to locate objects in real time more accurately.

[0066] When the image acquisition device is powered off, control the image acquisition device to turn on and acquire multiple consecutive image frames captured in real time as a real-time image sequence. In order to save energy, the image acquisition device may not always be in shooting mode. When the terminal receives the image sequence acquisition instruction, it can turn on the image acquisition device to start shooting and use the captured consecutive image frames as a real-time image sequence.

[0067] The real-time image sequence is continuously updated as the image acquisition device captures images. As time progresses, the terminal can continuously acquire new image frames as the current image frame through the image acquisition device, resulting in a real-time image sequence that is updated frame by frame.

[0068] Step S2: Obtain the dynamic background image based on the real-time image sequence.

[0069] The dynamic background image can be obtained based on each consecutive image frame in the real-time image sequence; specifically, methods such as the median method, the mean method, and the Kalman filter model can be used to obtain the dynamic background image corresponding to the real-time image sequence.

[0070] In practice, as the image acquisition device continuously acquires new images, the number of consecutive image frames in the real-time image sequence also increases, meaning the real-time image sequence is constantly updated. The dynamic background image also updates dynamically as the real-time image sequence changes, thus reducing interference caused by changes in the external environment.

[0071] Step S3: Obtain the difference map between every two adjacent image frames in the real-time image sequence, and obtain the average difference map based on each difference map.

[0072] The real-time image sequence includes frame 1, frame 2, frame 3, ..., frame t and frame t+1. Two adjacent image frames can be frame 1 and frame 2, frame 2 and frame 3, ..., frame t-1 and frame t, frame t and frame t+1, or frame 1 and frame 2, frame 3 and frame 4, ..., frame t-2 and frame t-1, frame t and frame t+1, where t is an integer value. The average difference map is an image obtained by averaging all difference maps, and it is also dynamically updated as the real-time image sequence changes.

[0073] Specifically, the terminal can calculate the difference map between every two adjacent image frames in the real-time image sequence, calculate the average value of the multiple difference maps, and obtain the average difference map corresponding to the multiple difference maps.

[0074] Step S4: Obtain the sum of pixel values ​​in the difference map between the current image frame and the previous image frame in the real-time image sequence, and calculate the current difference map between the current image frame and the dynamic background map when the sum of pixel values ​​is less than a preset threshold.

[0075] In this real-time image sequence, the current image frame is the latest image frame acquired by the image acquisition device. Since the more similar two images are, the smaller the sum of pixel values ​​in their difference images, when the sum of pixel values ​​approaches zero, it indicates that the current image is almost static, which can be considered as the object being placed in the object detection area. At this point, acquiring new image frames can be stopped. In practical implementation, the preset threshold can be set to a value very close to zero, such as 0.001 or 0.002.

[0076] Specifically, the terminal can obtain the difference map between the latest acquired current image frame and its previous image frame from step S3, and then calculate the sum of pixel values ​​in the difference map. If the sum of pixel values ​​is less than a preset threshold, it is considered that the sum of pixel values ​​is close to or nearly equal to zero. At this time, the current image frame can be considered as the image frame that needs to be located, and there is no need to acquire a new image frame. The terminal can also calculate the current difference map between the current image frame and the latest dynamic background image to perform preliminary location of the item in the current image frame.

[0077] In one embodiment of this invention, the method may further include the following steps:

[0078] When the sum of pixel values ​​in the difference map between the current image frame and the previous image frame in the real-time image sequence is less than a preset threshold, the acquisition of new image frames is stopped.

[0079] In the above implementation, when the difference between the current and subsequent frames is small, image updates can be automatically stopped, and the latest image frame can be selected for item positioning without the need for manual operation to specify the recognition frame.

[0080] Step S5: Perform dot product and inversion calculations on the current difference map based on the average difference map to obtain the item location map, and determine the connected region with the largest area in the item location map as the item region location.

[0081] The item location map is the result of multiplying and inverting the average difference map and the current difference map. The average difference map is used as a weight map and multiplied and inverted with the current difference map obtained in step S4. This can reduce the interference to the background caused by the movement of the item, thus obtaining an item location map that is less sensitive to changes in the background.

[0082] Specifically, the connected components in the object location map are identified. Usually, more than one connected component can be obtained. The connected component with the largest area is determined as the location of the object region, thus completing the location of the object in the object detection area, which is the area to be identified in subsequent object recognition.

[0083] In one embodiment of this example, step S5 may include the following steps: performing threshold segmentation and connected component extraction on the item location map to determine each connected component in the item location map, and determining the connected component with the largest area among each connected component as the location of the item region.

[0084] Specifically, a binary image is obtained by thresholding the item location map. Then, Blob analysis is used to extract and label connected components, which yields information on all connected components in the item location map. The connected component with the largest area is selected as the item region location, i.e., the region where the item to be identified is located. In practical applications, subsequent item identification only requires identifying the image of the item region location, rather than the entire image, thus reducing the amount of data processed.

[0085] This application provides an object positioning method that can acquire a real-time image sequence of an object detection area through an image acquisition device. The real-time image sequence includes multiple consecutive image frames, and the real-time positioning of objects in the images is achieved based on the real-time image sequence. A dynamic background image is obtained based on the real-time image sequence. A difference image between every two adjacent image frames in the real-time image sequence is obtained, and an average difference image is obtained based on each difference image. Both the dynamic background image and the average difference image are dynamically updated as the real-time image sequence is updated. When the sum of pixel values ​​in the difference image between the current image frame and the previous image frame is less than a preset threshold, a current difference image between the current image frame and the dynamic background image is calculated. The current difference image is then multiplied and inverted based on the average difference image to obtain an object positioning image. The connected component with the largest area in the object positioning image is determined as the location of the object region. The above method does not require a pre-set background image, but dynamically updates the background image during the localization process, avoiding interference from changes in the external environment. At the same time, by using the average difference map as a weight map and performing dot product and inversion calculations with the current difference map, the interference caused by the object's movement on the background can be further reduced, thereby improving the accuracy of localization. In addition, compared with object detection algorithms, the above method can perform detection without relying on product features, has better versatility, can still complete localization on new items, and reduces the consumption of computing resources.

[0086] Based on the above embodiments, in some embodiments, step S2 may include the following steps:

[0087] The dynamic background image is obtained by calculating the average value of all image frames in the real-time image sequence.

[0088] The current frame can be denoted as frame t+1, meaning that the real-time image sequence includes t+1 consecutive image frames. The dynamic background image is then determined based on the average value of these t+1 consecutive image frames.

[0089] Specifically, the terminal calculates the average value of all consecutive image frames t+1 in the real-time image sequence to obtain the dynamic background image. Furthermore, the image acquisition device calculates the average value of the latest image frame and all previous image frames for each frame captured, thereby achieving frame-by-frame updating of the dynamic background image.

[0090] In the above embodiments, the method uses the mean method to obtain the dynamic background image, which has a faster calculation speed and can improve the efficiency of obtaining the dynamic background image. Furthermore, since the above method does not require a preset background image, but dynamically updates the background image during image recognition, it can avoid interference caused by changes in the external environment, thereby improving the accuracy of item positioning.

[0091] Please see Figure 2 In some embodiments, step S3 may include the following steps:

[0092] Step S31: Preprocess every two adjacent image frames in the real-time image sequence to obtain multiple pairs of preprocessed adjacent image frames.

[0093] Preprocessing can involve scaling and / or noise reduction of each image frame to reduce the amount of data processed and / or improve image frame quality. In practice, the scaling degree of each image frame can be determined by the user according to actual needs; for example, scaling can be performed on each image frame while ensuring the edges of the object remain sharp.

[0094] Step S32: The difference map of each pair of preprocessed adjacent image frames is calculated by the structural similarity algorithm, and the average difference map is obtained based on each difference map.

[0095] Structural Similarity (SSIM) is a metric for measuring the similarity between two images. The SSIM index defines structural information from the perspective of image composition as an attribute that is independent of brightness and contrast, reflecting the structure of objects in a scene. It models distortion as a combination of three different factors: brightness, contrast, and structure. The mean is used to estimate brightness, the standard deviation to estimate contrast, and the covariance to measure the degree of structural similarity. The SSIM algorithm is better suited for extracting structural information, and the difference map calculated using it can better reflect the structural differences between adjacent image frames, making it suitable for applications such as unmanned shelves and smart retail cabinets.

[0096] In the above embodiments, the method can reduce the amount of data to be processed by image preprocessing and improve the efficiency of subsequent item positioning. At the same time, a structural similarity algorithm is used to calculate the difference map between adjacent image frames. Since the structural similarity algorithm has good adaptability to capturing structural information, it can obtain a difference map that more accurately reflects the differences, which is beneficial to improving the accuracy of subsequent item positioning.

[0097] Based on the above embodiments, in one implementation, step S31 may include the following steps: scaling each image frame in every two adjacent image frames to obtain multiple pairs of scaled adjacent image frames.

[0098] Step S32 may include the following steps: calculating the difference map of each pair of scaled adjacent image frames using a structural similarity algorithm, and obtaining an average difference map based on each difference map, wherein the size of the average difference map is the scaled size.

[0099] Step S5 may include the following steps: restoring the average difference map to the size of the current image frame as a weight map; performing dot product and inversion calculations on the weight map and the current difference map to obtain the item location map.

[0100] In the above embodiments, the method can obtain a small-sized average difference map by scaling the image frame. Then, when calculating the object location map, the small-sized average difference map needs to be restored to the same size as the current image frame to be used as a weight map for dot product and inversion calculations. Using the average difference map as a weight map can reduce the interference caused to the background during the movement of the object. At the same time, since the scaled image frame is used when calculating the average difference map, the amount of data for difference map calculation is reduced, which can improve the calculation efficiency.

[0101] Based on the above embodiments, in one implementation, step S32 may include the following steps:

[0102] Calculate the mean matrix and variance matrix of each image frame in each pair of preprocessed adjacent image frames, and calculate the covariance matrix between each pair of preprocessed adjacent image frames.

[0103] Specifically, the mean matrix of each image frame is calculated, and then the difference matrix of each image frame is calculated based on the mean matrix of each image frame. Then, the variance matrix of each image frame is calculated based on the mean matrix and the difference matrix of each image frame. Finally, the difference matrices of two adjacent preprocessed image frames are multiplied by a dot product. The difference dot product matrix and the mean matrices of the two adjacent image frames are then processed to obtain the covariance matrix between the two adjacent preprocessed image frames.

[0104] Based on the mean and variance matrices of each image frame in each pair of preprocessed adjacent image frames, and the covariance matrix between each pair of preprocessed adjacent image frames, a difference map of each pair of preprocessed adjacent image frames is obtained based on the structural similarity calculation formula.

[0105] In this embodiment, x and y represent two adjacent image frames, and the structural similarity calculation formula is abbreviated as the SSIM calculation formula. The difference image is shown below:

[0106]

[0107] Where 1 = 5 and c2 = 45.

[0108] Specifically, the mean matrix μ of image frame x is calculated according to the SSIM formula. x and variance matrix σ x 2 The mean matrix μ of image frame y y and variance matrix σ y 2 And the covariance matrix σ between image frame x and image frame y xy These parameters are calculated to obtain the difference map SSIM(x,y) between image frame x and image frame y.

[0109] In some embodiments, step S4, calculating the current difference map between the current image frame and the dynamic background image, may include the following steps: calculating the current difference map between the current image frame and the dynamic background image using a structural similarity algorithm.

[0110] The current image frame is the latest image frame acquired by the image acquisition device, and the dynamic background image is the latest dynamic background image corresponding to the current image frame.

[0111] Specifically, the terminal can use a structural similarity algorithm to calculate the current difference map between the current image frame and the dynamic background map. Since the structural similarity algorithm used in step S4 is the same as that in step S32, it will not be described again here.

[0112] In the above embodiments, the method uses a structural similarity algorithm to calculate the current difference map between the current image frame and the dynamic background image. Since the structural similarity algorithm is highly adaptable to capturing structural information, it can obtain a difference map that more accurately reflects the differences, which is beneficial to improving the accuracy of subsequent item positioning.

[0113] Please see Figure 3 One embodiment of this application provides an item identification method, the method comprising:

[0114] Step S1: Acquire a real-time image sequence of the object detection area using an image acquisition device. The real-time image sequence includes multiple consecutive image frames.

[0115] Step S2: Obtain the dynamic background image based on the real-time image sequence.

[0116] Step S3: Obtain the difference map between every two adjacent image frames in the real-time image sequence, and obtain the average difference map based on each difference map.

[0117] Step S4: Obtain the sum of pixel values ​​in the difference map between the current image frame and the previous image frame in the real-time image sequence, and calculate the current difference map between the current image frame and the dynamic background map when the sum of pixel values ​​is less than a preset threshold.

[0118] Step S5: Perform dot product and inversion calculations on the current difference map based on the average difference map to obtain the item location map, and determine the connected region with the largest area in the item location map as the item region location.

[0119] The item location refers to the area where the item to be identified is located. In practice, the item to be identified can be goods, materials, etc., that have been picked up or placed.

[0120] Step S6: The image of the location of the object region is identified by the trained deep learning neural network to obtain the identification result of the object to be identified.

[0121] The object recognition result can include information such as the type of the object to be identified and the quantity of objects taken or placed. The deep learning neural network can be trained based on a set of object sample images. In specific implementations, the aforementioned deep learning neural network can be a commonly used object recognition network structure in existing technologies, such as CNN, FCN, YOLO, or SSD.

[0122] The object recognition method provided in the above embodiments does not require a preset background image during the object localization process, but dynamically updates the background image to avoid interference caused by changes in the external environment. At the same time, by using the average difference image as a weight image and performing dot product and inversion calculations with the current difference image, the interference caused to the background during the movement of the object can be further reduced, resulting in a more accurate object area location and helping to improve the accuracy of object recognition.

[0123] See Figure 4 One embodiment of this application provides an article positioning device, which includes:

[0124] The image sequence acquisition module 101 is used to acquire a real-time image sequence of the object detection area through the image acquisition device. The real-time image sequence includes multiple consecutive image frames.

[0125] The dynamic background image calculation module 102 is used to obtain a dynamic background image based on a real-time image sequence.

[0126] The average difference map calculation module 103 is used to obtain the difference map between every two adjacent image frames in the real-time image sequence, and to obtain the average difference map based on each difference map.

[0127] The current difference map calculation module 104 is used to obtain the sum of pixel values ​​in the difference map between the current image frame and the previous image frame in the real-time image sequence, and to calculate the current difference map between the current image frame and the dynamic background map when the sum of pixel values ​​is less than a preset threshold.

[0128] The item location module 105 is used to perform dot product and inversion calculations on the current difference map based on the average difference map to obtain the item location map, and to determine the connected region with the largest area in the item location map as the item region location.

[0129] In some embodiments, the dynamic background image calculation module 102 is specifically used to: calculate the average value of all image frames in a real-time image sequence to obtain a dynamic background image.

[0130] In some embodiments, the average difference plot calculation module 103 may include:

[0131] The preprocessing unit is used to preprocess every two adjacent image frames in the real-time image sequence to obtain multiple pairs of preprocessed adjacent image frames.

[0132] The average difference map calculation unit is used to calculate the difference map of each pair of preprocessed adjacent image frames through a structural similarity algorithm, and to obtain the average difference map based on each difference map.

[0133] In some embodiments, the preprocessing unit is specifically used to: scale each image frame in every two adjacent image frames to obtain multiple pairs of scaled adjacent image frames.

[0134] The average difference map calculation unit is specifically used to: calculate the difference map of each pair of scaled adjacent image frames through a structural similarity algorithm, and obtain the average difference map based on each difference map. The size of the average difference map is the scaled size.

[0135] The item location module 105 is specifically used to: restore the average difference map to the size of the current image frame as a weight map; and perform dot product and inversion calculations between the weight map and the current difference map to obtain the item location map.

[0136] In some embodiments, the average difference map calculation unit is specifically used to: calculate the mean matrix and variance matrix of each image frame in each pair of preprocessed adjacent image frames, and calculate the covariance matrix between each pair of preprocessed adjacent image frames; and obtain the difference map of each pair of preprocessed adjacent image frames based on the structural similarity calculation formula, according to the mean matrix and variance matrix of each image frame in each pair of preprocessed adjacent image frames and the covariance matrix between each pair of preprocessed adjacent image frames.

[0137] In some embodiments, the current difference map calculation module 104 is specifically used to: calculate the current difference map between the current image frame and the dynamic background map through a structural similarity algorithm.

[0138] In some embodiments, the item location module 105 is specifically used to: perform threshold segmentation and connected component extraction on the item location map to determine each connected component in the item location map, and determine the connected component with the largest area among each connected component as the location of the item region.

[0139] In some embodiments, the image sequence acquisition module 101 is specifically configured to: detect the current state of the image acquisition device in response to the image sequence acquisition instruction; when the image acquisition device is in the shooting state, acquire the current image frame captured by the image acquisition device and a number of consecutive image frames preceding the current image frame as a real-time image sequence; when the image acquisition device is in the off state, control the image acquisition device to turn on and acquire a number of consecutive image frames captured by the image acquisition device in real time as a real-time image sequence; the real-time image sequence is continuously updated as the image acquisition device continues to shoot.

[0140] In some embodiments, the apparatus further includes an image stop acquisition module, configured to stop acquiring a new image frame when the sum of pixel values ​​in the difference map between the current image frame and the previous image frame in the real-time image sequence is less than a preset threshold.

[0141] The specific limitations of the item positioning device provided in this embodiment can be found in the embodiments of the item positioning method described above, and will not be repeated here. Each module in the above-described item positioning device can be implemented entirely or partially through software, hardware, or a combination thereof. Each module can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0142] Please see Figure 5 One embodiment of this application provides an item identification device, which includes an item identification module 106 and an item positioning device as described in any of the above embodiments, wherein the original item area position is the area position where the item to be identified is located.

[0143] The item recognition module 106 is used to recognize the image of the location of the item area through a trained deep learning neural network to obtain the recognition result of the item to be recognized.

[0144] The specific limitations of the item recognition device provided in this embodiment can be found in the embodiments of the item recognition method described above, and will not be repeated here. Each module in the above-described item recognition device can be implemented entirely or partially through software, hardware, or a combination thereof. Each module can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0145] One embodiment of this application provides an intelligent vending machine, including a processor, a memory, and an image acquisition device;

[0146] The image acquisition device is used to acquire real-time image sequences of the object detection area;

[0147] The memory is used to store computer programs. When the smart vending machine is running, the processor executes the computer programs to implement the steps of the item positioning method or item identification method as described in any of the above embodiments.

[0148] One embodiment of this application provides an electronic device that may include a processor, a memory, a network interface, and a database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the processor executes the computer program, it implements the steps of the item location method or item identification method as described in any of the above embodiments.

[0149] The working process, working details and technical effects of the electronic device provided in this embodiment can be found in the embodiments of the item positioning method or item identification method mentioned above, and will not be repeated here.

[0150] One embodiment of this application provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the steps of the item positioning method or item identification method as described in any of the above embodiments. The computer-readable storage medium refers to a data storage medium, which may include, but is not limited to, floppy disks, optical disks, hard disks, flash memory, USB flash drives, and / or memory sticks. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0151] The working process, working details, and technical effects of the computer-readable storage medium provided in this embodiment can be found in the embodiments of the item positioning method or item identification method described above, and will not be repeated here.

[0152] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0153] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0154] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for locating an item, characterized in that, The method includes: A real-time image sequence of the object detection area is acquired by an image acquisition device, the real-time image sequence comprising multiple consecutive image frames; A dynamic background image is obtained based on the real-time image sequence; Obtain the difference map between every two adjacent image frames in the real-time image sequence, and obtain the average difference map based on each difference map; Obtain the sum of pixel values ​​in the difference map between the current image frame and the previous image frame in the real-time image sequence, and when the sum of pixel values ​​is less than a preset threshold, calculate the current difference map between the current image frame and the dynamic background image; The current difference map is multiplied and inverted based on the average difference map to obtain the item location map. The connected region with the largest area in the item location map is determined as the location of the item region.

2. The method according to claim 1, characterized in that, The step of obtaining the dynamic background image based on the real-time image sequence includes: The dynamic background image is obtained by calculating the average value of all image frames in the real-time image sequence.

3. The method according to claim 1, characterized in that, The step of obtaining the difference map between every two adjacent image frames in the real-time image sequence, and obtaining the average difference map based on each difference map, includes: Preprocessing is performed on every two adjacent image frames in the real-time image sequence to obtain multiple pairs of preprocessed adjacent image frames; The difference map of each pair of preprocessed adjacent image frames is calculated using a structural similarity algorithm, and the average difference map is obtained based on each difference map.

4. The method according to claim 3, characterized in that, The step of preprocessing every two adjacent image frames in the real-time image sequence to obtain multiple pairs of preprocessed adjacent image frames includes: Each image frame in every two adjacent image frames is scaled to obtain multiple pairs of scaled adjacent image frames. The step of calculating the difference map of each pair of preprocessed adjacent image frames using a structural similarity algorithm, and obtaining the average difference map based on each difference map, includes: The difference map of each pair of scaled adjacent image frames is calculated by a structural similarity algorithm, and an average difference map is obtained based on each difference map. The size of the average difference map is the scaled size. The step of performing dot product and inversion calculations on the current difference map based on the average difference map to obtain the item location map includes: The average difference map is restored to the size of the current image frame to serve as a weight map; The item location map is obtained by performing a dot product and inversion between the weight map and the current difference map.

5. The method according to claim 3, characterized in that, The step of calculating the difference map for each pair of preprocessed adjacent image frames using a structural similarity algorithm includes: Calculate the mean matrix and variance matrix of each image frame in each pair of preprocessed adjacent image frames, and calculate the covariance matrix between each pair of preprocessed adjacent image frames; Based on the mean matrix and variance matrix of each image frame in each pair of preprocessed adjacent image frames, and the covariance matrix between each pair of preprocessed adjacent image frames, a difference map of each pair of preprocessed adjacent image frames is obtained based on the structural similarity calculation formula.

6. The method according to claim 1, characterized in that, The calculation of the current difference map between the current image frame and the dynamic background map includes: The current difference map between the current image frame and the dynamic background map is calculated using a structural similarity algorithm.

7. The method according to claim 1, characterized in that, Determining the location of the item region as the connected region with the largest area in the item location map includes: Threshold segmentation and connected component extraction are performed on the item location map to determine each connected component in the item location map, and the connected component with the largest area among all connected components is determined as the item region location.

8. The method according to any one of claims 1 to 7, characterized in that, The process of acquiring a real-time image sequence of the object detection area via an image acquisition device includes: In response to an image sequence acquisition command, the current state of the image acquisition device is detected; When the image acquisition device is in shooting mode, the current image frame captured by the image acquisition device and a preset number of consecutive image frames before the current image frame are acquired as a real-time image sequence. When the image acquisition device is in the off state, control the image acquisition device to turn on and acquire multiple consecutive image frames captured by the image acquisition device in real time as a real-time image sequence; The real-time image sequence is continuously updated as the image acquisition device continues to capture images.

9. The method according to claim 8, characterized in that, The method further includes: When the sum of pixel values ​​in the difference map between the current image frame and the previous image frame in the real-time image sequence is less than a preset threshold, the acquisition of new image frames is stopped.

10. An item recognition method, characterized in that, The method includes the item location method as described in any one of claims 1 to 9, wherein the item region location is the region location where the item to be identified is located; The method further includes: identifying the image of the location of the object region using a trained deep learning neural network to obtain the identification result of the object to be identified.

11. An item positioning device, characterized in that, The device includes: An image sequence acquisition module is used to acquire a real-time image sequence of the object detection area through an image acquisition device, wherein the real-time image sequence includes multiple consecutive image frames; A dynamic background image calculation module is used to obtain a dynamic background image based on the real-time image sequence; The average difference map calculation module is used to obtain the difference map between every two adjacent image frames in the real-time image sequence, and to obtain the average difference map based on each difference map; The current difference map calculation module is used to obtain the sum of pixel values ​​in the difference map between the current image frame and the previous image frame in the real-time image sequence, and to calculate the current difference map between the current image frame and the dynamic background map when the sum of pixel values ​​is less than a preset threshold. The item location module is used to perform dot product and inversion calculations on the current difference map based on the average difference map to obtain the item location map, and to determine the connected region with the largest area in the item location map as the item region location.

12. An item identification device, characterized in that, The device includes an item recognition module and an item positioning device as described in claim 11, wherein the item region location is the region location where the item to be identified is located. The item recognition module is used to recognize the image of the item region location through a trained deep learning neural network to obtain the recognition result of the item to be recognized.

13. A smart vending machine, characterized in that, Includes a processor, memory, and image acquisition device; The image acquisition device is used to acquire real-time image sequences of the object detection area; The memory is used to store a computer program, and when the smart vending machine is running, the processor executes the computer program to implement the steps of the method as described in any one of claims 1 to 10.

14. An electronic device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 10.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 10.

Images