Commodity interest identification method based on visual algorithm

Through the product interest recognition method based on visual algorithm, the pre-trained object detection model and feature extraction algorithm are used, combined with CIoU loss function and trajectory merging technology, the problem of insufficient adaptability of complex scenes in the existing technology is solved, and automatic recognition is achieved The functions of products that customers are interested in improve shopping guide efficiency and save costs.

CN119027646APending Publication Date: 2024-11-26SHANGHAI ULUCU ELECTRON TECH CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411056799.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-02
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The prior art has the disadvantages of insufficient adaptability to complex scenarios and obvious distinction between targets and backgrounds in the identification of product interest, resulting in low shopping guide efficiency and high cost.

Method used

Using the product interest recognition method based on visual algorithm, the monitoring video is obtained through the pre-trained object detection model, the target tracking and attribute determination are performed, the detection model is adjusted in combination with the CIoU loss function, the features are extracted using ShuffleNetV2, and the trajectory is tracked through the SORT algorithm, and the trajectory is merged. Use ReID and ResNet18 to determine the interest of pedestrians in the commodity area.

Benefits of technology

It realizes automatic identification of customers' interest in products, improves the accuracy and efficiency of shopping guides, saves labor costs, and pushes information to shopping guides to provide accurate shopping guides.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027646A_ABST
    Figure CN119027646A_ABST
Patent Text Reader

Abstract

The invention provides a commodity interest recognition method based on a visual algorithm, and belongs to the technical field of object recognition. The method comprises the following steps: obtaining a monitoring video in a target scene, and obtaining a target detection object by using a pre-trained target detection model; performing target tracking on the target detection object to obtain a tracking trajectory, and determining attributes (sitting, lying and standing) of the target detection object according to the tracking trajectory; and based on the attribute of the target detection object, comparing the tracking trajectory with a preset commodity area, and if the coincidence duration exceeds a threshold value, judging that the pedestrian is interested in the commodity area.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of object recognition, and in particular to a method for identifying commodities of interest based on a visual algorithm. Background Art

[0002] In large shopping malls, there are many merchandise showrooms, and each showroom is usually equipped with corresponding service guides. After customers enter the showroom, the guides will follow the customers who come to view the merchandise through multiple showrooms. This will reduce the efficiency of the guides, and in turn will lead to the need to recruit a large number of guides.

[0003] At present, with the development and progress of computer vision technology and machine learning, it is becoming more and more possible to automatically identify events through monitoring, such as pedestrian detection, face recognition, etc. In the existing technology, the use of computer vision algorithms to automatically identify the products that customers are interested in and the corresponding exhibition halls can not only improve the efficiency of shopping guides, improve the accuracy of customer shopping guides, but also save the employment costs of shopping guides.

[0004] Existing detection technologies can be divided into two categories:

[0005] (1) Traditional moving target detection technologies, such as frame difference method, background difference method, optical flow method, etc., are characterized by simple calculation and strong real-time performance. However, they have the disadvantage of being unable to cope with complex scenes.

[0006] (2) Object detection technology based on deep learning, represented by Faster-RCNN, YOLO, SSD, etc., is characterized by strong adaptability to complex scenes. However, it has the disadvantage that the target to be detected and the background have obvious distinguishable features.

[0007] Therefore, it is urgent to design a product interest identification method based on visual algorithms to solve the above problems. Summary of the invention

[0008] The purpose of the present invention is to overcome the deficiencies of the above-mentioned prior art and provide a method for identifying products of interest based on a visual algorithm.

[0009] To achieve the above objectives, the present disclosure adopts the following technical solutions:

[0010] According to one aspect of the present disclosure, a method for identifying products of interest based on a visual algorithm is provided, the method comprising the following steps:

[0011] Obtain surveillance video of the target scene and use the pre-trained target detection model to obtain the target detection object;

[0012] Tracking the target detection object to obtain a tracking trajectory, and determining the attribute (sitting, lying, standing) of the target detection object according to the tracking trajectory;

[0013] Based on the attributes of the target detection object, the tracking trajectory is compared with a preset commodity area. If the overlap time exceeds a threshold, it is determined that the pedestrian is interested in the commodity area.

[0014] In an exemplary embodiment of the present disclosure, the training process of the target detection model includes:

[0015] Extracting framed images from the surveillance video, and selecting image frames with the target detection object as training data;

[0016] On the image frame with the target detection object, manually marking the target detection object frame and the target detection object attributes;

[0017] The labeled data is mixed with the coco public dataset to train the object detection model.

[0018] In an exemplary embodiment of the present disclosure, the training process of the target detection model further includes:

[0019] The target detection object is detected using the general target detection algorithm yolov4, and the features of the target detection object are extracted through the target detection neural network ShuffleNetV2 in the general target detection algorithm.

[0020] In an exemplary embodiment of the present disclosure, the general target detection algorithm is set on a camera with a chip, and the feature extraction neural network ShuffleNetV2 in the target detection includes a convolution layer, a pooling layer, 3 groups of residual modules and a convolution, and the outputs of the first two groups of residual modules and the last convolution are used as feature outputs.

[0021] In an exemplary embodiment of the present disclosure, detecting the target detection object using the general target detection algorithm yolov4 includes:

[0022] Based on the training data, the general target detection algorithm yolov4 is trained;

[0023] Using CIoU as the loss function, the parameters of the general target detection algorithm yolov4 are adjusted to obtain the target detection model.

[0024] In an exemplary embodiment of the present disclosure, the loss function CIoU is expressed as:

[0025]

[0026] in,

[0027]

[0028] b,b gt Represent the center points of the anchor box and the target box respectively, ρ(·) represents the Euclidean distance, c represents the diagonal distance of the smallest rectangle that can cover both the anchor box and the target box, A represents the anchor box, B represents the target box, ω, h represent the width and height of the prediction box respectively, ω gt ,h gt are the width and height of the real frame respectively.

[0029] In an exemplary embodiment of the present disclosure, the target tracking of the target detection object includes:

[0030] The multiple detected target detection objects are tracked by using the SORT algorithm.

[0031] In an exemplary embodiment of the present disclosure, obtaining the tracking trajectory includes:

[0032] During the target detection object tracking process, a tracking track is generated, and at an interval of 60 seconds, it is determined whether there is a track that needs to be merged in the existing tracking track;

[0033] If it exists, the feature information of the target detection object is extracted through the HA-CNN used in ReID, and the feature information includes global features and local features;

[0034] The trajectories with the same feature information are merged and connected through the ReID.

[0035] In an exemplary embodiment of the present disclosure, before determining the attribute of the target detection object according to the tracking trajectory, the method further includes selecting a human-shaped picture in the tracking trajectory,

[0036] The selecting a human figure in the tracking trajectory comprises:

[0037] Tracking the detected target detection object using a target tracking algorithm;

[0038] caching a certain number of target images in the tracking trajectory;

[0039] According to the human figure related information, an image close to the image center and having a higher target detection object score is selected from the cached target images.

[0040] In an exemplary embodiment of the present disclosure, determining the attribute of the target detection object according to the tracking trajectory includes:

[0041] The attributes of the target detection object are trained using the training data and the labeled data through the image classification network ResNet18.

[0042] In an exemplary embodiment of the present disclosure, before comparing the tracking trajectory with a preset commodity area, the method further includes:

[0043] Divide the commodity area into regions, and record the commodity area as Q;

[0044] The commodity area can be expressed as:

[0045] Q={Q1,Q2,Q3,…,Qi,…,Qn},

[0046] Among them, n represents the number of commodity areas, and Qi,represents the i-th commodity area.

[0047] In an exemplary embodiment of the present disclosure, comparing the tracking trajectory with a preset commodity area includes:

[0048] Counting the overlap time between the tracking track and the commodity area;

[0049] If the overlap duration is longer than 15 milliseconds, it is determined that the pedestrian is interested in the commodity area;

[0050] The commodity area that the pedestrian is interested in, the length of time the pedestrian stays in the commodity area, and the attribute status of the pedestrian during the length of time are recorded.

[0051] In an exemplary embodiment of the present disclosure, the method further includes:

[0052] After the pedestrian enters the exhibition hall, the system pushes the merchandise area that the pedestrian is interested in to the shopping guide.

[0053] The present invention has at least the following beneficial effects:

[0054] The present invention provides a method for identifying products of interest based on a visual algorithm. When a customer enters an exhibition hall, the customer is automatically tracked by a monitoring camera to form a tracking trajectory. The products that the customer is interested in are determined based on the length of time that the tracking trajectory overlaps with a product area. At the same time, the customer's body attributes in the product area are recorded, such as sitting, lying or standing, to determine the customer's interest level in the product. A picture and product information that the customer is interested in are automatically saved on the tracking trajectory. The system pushes this information to a shopping guide's device. The shopping guide can browse the customer's interested information prompts to accurately guide shopping, save time and improve shopping guide efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0056] Figure 1 The present invention is a flowchart of a method for identifying products of interest based on a visual algorithm in one embodiment of the present invention. DETAILED DESCRIPTION

[0057] Example embodiments will now be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in a variety of forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that the present disclosure will be comprehensive and complete and fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar structures, and thus their detailed description will be omitted. In addition, the drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale.

[0058] The present disclosure provides a method for identifying products of interest based on a visual algorithm. Figure 1 As shown, the method for identifying commodities of interest based on a visual algorithm includes the following steps:

[0059] Step 101, obtain a surveillance video of a target scene, and obtain a target detection object using a pre-trained target detection model.

[0060] In this embodiment, several surveillance videos of target scenes are collected from all surveillance videos to train the target detection model. Specifically, the surveillance videos may be surveillance videos in a hall or surveillance videos in an exhibition hall.

[0061] In one embodiment, the training process of the target detection model includes:

[0062] Extract framed images from the surveillance video, and select image frames with target detection objects as training data;

[0063] On the image frame with the target detection object, manually mark the target detection object frame and the target detection object attributes;

[0064] Mix the labeled data with the coco public dataset to train the object detection model.

[0065] In actual use, several surveillance videos of target scenes are collected and framed images are extracted, and image frames with target detection objects are selected as training data, where the target detection objects are pedestrians. For the selected image frames, the target detection object boxes and target detection object attributes are manually annotated, that is, the pedestrian boxes and human attributes are manually annotated. Human attributes include sitting, lying, and standing. The annotated pedestrian boxes and human attributes are mixed with the coco public dataset, and the mixed dataset is used to train the target detection model.

[0066] In one embodiment, the training process of the target detection model further includes:

[0067] The general target detection algorithm yolov4 is used to detect the target detection object, and the features of the target detection object are extracted through the feature extraction neural network ShuffleNetV2.

[0068] In one embodiment, a general target detection algorithm is set on a camera with a chip, and the target detection neural network ShuffleNetV2 includes a convolutional layer, a pooling layer, 3 groups of residual modules and a convolution, and the outputs of the first two groups of residual modules and the last convolution are used as feature outputs.

[0069] In this embodiment, a general target detection algorithm is deployed on a camera with a chip. The original base network in the general target detection algorithm yolov4 is replaced by a light base network ShuffleNetV2 to extract the features of the target detection object. The light base network ShuffleNetV2 is mainly composed of a convolutional layer, a pooling layer, 3 groups of residual modules and a convolution. The outputs of two groups of residual modules and the last convolution are selected as feature outputs.

[0070] In one embodiment, detecting a target detection object using the general target detection algorithm yolov4 includes:

[0071] Based on the training data, the general target detection algorithm yolov4 is trained;

[0072] Using CIoU as the loss function, the parameters of the general target detection algorithm yolov4 are adjusted to obtain the target detection model.

[0073] In actual use, the selected training data is used to train the general target detection algorithm yolov4. The main loss function used by the general target detection algorithm yolov4 is CIoU. CIoU is used to adjust the parameters of the general target detection algorithm yolov4 to obtain a suitable target detection model and select a suitable target detection object.

[0074] In one embodiment, the loss function CIoU is expressed as:

[0075]

[0076] in,

[0077]

[0078] b,b gt Represent the center points of the anchor box and the target box respectively, ρ(·) represents the Euclidean distance, c represents the diagonal distance of the smallest rectangle that can cover both the anchor box and the target box, A represents the anchor box, B represents the target box, ω, h represent the width and height of the prediction box respectively, ω gt ,h gt are the width and height of the real frame respectively.

[0079] Step 102, tracking the target detection object to obtain a tracking trajectory, and determining the attribute of the target detection object (sitting, lying, standing) according to the tracking trajectory;

[0080] In this embodiment, multiple target detection objects are tracked simultaneously to obtain a tracking trajectory corresponding to each target detection object, and changes in attribute states of the target detection objects are obtained based on changes in the tracking trajectory. The attributes of the target detection objects include sitting, lying, and standing.

[0081] In one embodiment, tracking a target detection object includes:

[0082] Multiple detected target objects are tracked using the SORT algorithm.

[0083] Specifically, the multiple detected pedestrian frames are tracked using the SORT algorithm.

[0084] In one embodiment, obtaining the tracking trajectory includes:

[0085] During the process of target detection object tracking, tracking trajectories are generated, and at intervals of 60 seconds, it is determined whether there are trajectories that need to be merged in the existing tracking trajectories;

[0086] If it exists, the feature information of the target detection object is extracted through the HA-CNN used in ReID, and the feature information includes global features and local features;

[0087] ReID is used to merge and connect trajectories with the same feature information.

[0088] In actual use, during the pedestrian tracking process, each pedestrian will generate a tracking track. The tracking track will be interrupted due to object occlusion or crossing areas. The two tracking tracks will be compared at an interval of 60 seconds to determine whether they can be merged. If they can be merged, the HA-CNN used in ReID will be used to extract the global and local features of the target detection object. The two tracking tracks with the same feature information will be merged and connected through ReID to obtain a complete tracking track.

[0089] In one embodiment, before determining the attribute of the target detection object according to the tracking trajectory, the method further includes selecting a human-shaped image in the tracking trajectory,

[0090] Selecting a human figure in the tracking trajectory includes:

[0091] Track the detected target object using a target tracking algorithm;

[0092] Cache a certain number of target images in the tracking trajectory;

[0093] According to the human figure related information, an image close to the image center and having a higher target detection object score is selected from the cached target images.

[0094] In actual use, the detected human frame is tracked by the target tracking algorithm SORT to obtain a tracking trajectory. The tracking trajectory contains a certain number of target images. According to the human-related information, an image close to the center of the image and with a higher human frame score is selected from multiple target images. Among them, the human frame score is between 0.5 and 1. Among the multiple target images, the one with the highest score is selected to determine the human attributes.

[0095] In one embodiment, determining the attribute of the target detection object according to the tracking trajectory includes:

[0096] Using the training data and labeled data, the attributes of the target detection object are trained through the image classification network ResNet18.

[0097] Step 103 : Based on the attributes of the target detection object, the tracking trajectory is compared with the preset commodity area. If the overlap time exceeds a threshold, it is determined that the pedestrian is interested in the commodity area.

[0098] In actual use, the tracking trajectory that overlaps with the commodity area is selected according to the detected tracking trajectory and human body attributes, and the tracking trajectory is compared with the commodity area in the exhibition hall. If the overlap time of the tracking trajectory and the commodity area exceeds a certain threshold, it is judged that the pedestrian is interested in the commodity area. If the overlap time does not reach the threshold, it is judged that the pedestrian is not interested in the commodity area.

[0099] In one embodiment, before comparing the tracking trajectory with the preset commodity area, the method further includes:

[0100] Divide the commodity area into zones and record the commodity area as Q;

[0101] The commodity area can be expressed as:

[0102] Q={Q1,Q2,Q3,…,Qi,…,Qn},

[0103] Among them, n represents the number of commodity areas, and Qi,represents the i-th commodity area.

[0104] In actual use, the goods in the exhibition hall are divided into n areas in order according to their locations. The product area is represented by Q, and the i-th product area is represented by Qi, so as to determine which product the pedestrian is interested in.

[0105] In one embodiment, comparing the tracking trajectory with a preset commodity area includes:

[0106] Count the overlap time between the tracking track and the product area;

[0107] If the overlap duration is longer than 15 milliseconds, it is determined that the pedestrian is interested in the product area;

[0108] The merchandise area that the pedestrian is interested in, the length of time the pedestrian stays in the merchandise area, and the attribute status of the pedestrian during the length of time are recorded.

[0109] In actual use, when the tracking trajectory of a pedestrian overlaps with a certain product area, the overlapping time of the tracking trajectory and the product area is calculated. If the overlapping time is greater than 15 milliseconds, the pedestrian is determined to be interested in the product area. Then, the product area, the length of time the pedestrian stays, and the attribute status of the pedestrian during this time are recorded to determine the degree of the pedestrian's interest in the product.

[0110] Step 104: After the pedestrian enters the exhibition hall, the system pushes the merchandise area that the pedestrian is interested in to the shopping guide.

[0111] In actual use, when a customer enters an exhibition hall, the system will accurately push information about the product areas that the customer is interested in to the shopping guide's device. The shopping guide can browse the information about the product areas that the customer is interested in, provide accurate shopping guidance, save time, and improve shopping efficiency.

[0112] The method for identifying goods of interest based on a visual algorithm provided by an embodiment of the present invention first obtains a surveillance video of a target scene, and obtains a target detection object using a pre-trained target detection model; then, the target detection object is tracked to obtain a tracking trajectory, and the attributes of the target detection object (sitting, lying, standing) are determined based on the tracking trajectory; based on the attributes of the target detection object, the tracking trajectory is compared with a pre-set goods area, and if the overlap duration exceeds a threshold, it is determined that the pedestrian is interested in the goods area; after the pedestrian enters the exhibition hall, the system pushes the goods area of ​​interest to the shopping guide.

[0113] The present invention provides a method for identifying products of interest based on a visual algorithm. When a customer enters an exhibition hall, the customer is automatically tracked by a monitoring camera to form a tracking trajectory. The products that the customer is interested in are determined based on the length of time that the tracking trajectory overlaps with a product area. At the same time, the customer's body attributes in the product area are recorded, such as sitting, lying or standing, to determine the customer's interest level in the product. A picture and product information that the customer is interested in are automatically saved on the tracking trajectory. The system pushes this information to a shopping guide's device. The shopping guide can browse the customer's interested information prompts to accurately guide shopping, save time and improve shopping guide efficiency.

[0114] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A method for identifying products of interest based on visual algorithms, characterized in that: The method comprises the following steps: Obtain surveillance video of the target scene and use the pre-trained target detection model to obtain the target detection object; Tracking the target detection object to obtain a tracking trajectory, and determining the attribute (sitting, lying, standing) of the target detection object according to the tracking trajectory; Based on the attributes of the target detection object, the tracking trajectory is compared with a preset commodity area, and if the overlap time exceeds a threshold, it is determined that the pedestrian is interested in the commodity area; Comparing the tracking trajectory with the preset commodity area includes: Counting the overlap time between the tracking track and the commodity area; If the overlap duration is longer than 15 milliseconds, it is determined that the pedestrian is interested in the commodity area; The commodity area that the pedestrian is interested in, the length of time the pedestrian stays in the commodity area, and the attribute status of the pedestrian during the length of time are recorded.

2. The method for identifying products of interest based on visual algorithms according to claim 1, characterized in that: The training process of the target detection model includes: Extracting framed images from the surveillance video, and selecting image frames with the target detection object as training data; On the image frame with the target detection object, manually marking the target detection object frame and the target detection object attributes; Mix the labeled data with the coco public dataset to train the target detection model; The training process of the target detection model also includes: The target detection object is detected using the general target detection algorithm yolov4, and the features of the target detection object are extracted using the feature extraction neural network ShuffleNetV2. Determining the attribute of the target detection object according to the tracking trajectory includes: The attributes of the target detection object are trained using the training data and the labeled data through the image classification network ResNet18.

3. The method for identifying products of interest based on visual algorithms according to claim 2, characterized in that: The general target detection algorithm is set on a camera with a chip, and the feature extraction neural network ShuffleNetV2 includes a convolution layer, a pooling layer, 3 groups of residual modules and a convolution, and the outputs of the first two groups of residual modules and the last convolution are used as feature outputs.

4. The method for identifying products of interest based on visual algorithms according to claim 2, characterized in that: The detecting of the target detection object by using the general target detection algorithm yolov4 includes: Based on the training data, the general target detection algorithm yolov4 is trained; Using CIoU as the loss function, the parameters of the general target detection algorithm yolov4 are adjusted to obtain the target detection model.

5. The method for identifying products of interest based on visual algorithms according to claim 4, characterized in that: The loss function CIoU is expressed as: in, b,b gt Represent the center points of the anchor box and the target box respectively, ρ(·) represents the Euclidean distance, c represents the diagonal distance of the smallest rectangle that can cover both the anchor box and the target box, A represents the anchor box, B represents the target box, ω, h represent the width and height of the prediction box respectively, ω gt ,h gt are the width and height of the real frame respectively.

6. The method for identifying products of interest based on visual algorithms according to claim 1, characterized in that: The target tracking of the target detection object comprises: The multiple detected target detection objects are tracked by using the SORT algorithm.

7. The method for identifying products of interest based on visual algorithms according to claim 1, characterized in that: The obtaining of the tracking trajectory comprises: During the target detection object tracking process, a tracking track is generated, and at an interval of 60 seconds, it is determined whether there is a track that needs to be merged in the existing tracking track; If it exists, the feature information of the target detection object is extracted through the HA-CNN used in ReID, and the feature information includes global features and local features; The trajectories with the same feature information are merged and connected through the ReID.

8. The method for identifying products of interest based on visual algorithms according to claim 1, characterized in that: Before determining the attribute of the target detection object according to the tracking trajectory, the method further includes selecting a human-shaped picture in the tracking trajectory, The selecting a human figure in the tracking trajectory comprises: Tracking the detected target detection object using a target tracking algorithm; caching a certain number of target images in the tracking trajectory; According to the human figure related information, an image close to the image center and having a higher target detection object score is selected from the cached target images.

9. The method for identifying products of interest based on visual algorithms according to claim 1, characterized in that: Before comparing the tracking trajectory with the preset commodity area, the method further includes: Divide the commodity area into regions, and record the commodity area as Q; The commodity area can be expressed as: Q={Q1,Q2,Q3,…,Qi,…,Qn}, Among them, n represents the number of commodity areas, and Qi,represents the i-th commodity area.

10. The method for identifying products of interest based on visual algorithms according to claim 1, characterized in that: After the pedestrian enters the exhibition hall, the system pushes the merchandise area that the pedestrian is interested in to the shopping guide.

Citation Information

Patent Citations

  • Precision marketing evaluation system based on visual perception and operation data quantification

    CN112258243A

  • Tracking target state analysis method and device, storage medium and terminal

    CN114820692A

  • Commodity auxiliary identification method based on online retrieval learning and behavior track

    CN117078297A

  • Vision-based goods shelf commodity taking and placing behavior detection method

    CN117292323A

  • Person behavior analysis device, person behavior analysis system, person behavior analysis method, and monitoring device

    US20150010204A1