A method and system for detecting the operating state of a store, a terminal and a storage medium

Abstract

The application provides a store business state detection method and system, a terminal and a storage medium, and relates to the technical field of computers. The method comprises the following steps: obtaining a store scene image of a store to be detected; obtaining a pre-trained business state detection model, which is improved based on a YOLO network framework and comprises a backbone network, a neck network, a detection head and a state classification branch; inputting the store scene image into the business state detection model to obtain a first detection result and a second detection result, wherein the first detection result comprises the probability distribution of four types of set states, and the set states comprise the states of a light-on door and window, a light-on door and window, a light-off door and window and a light-off door and window; the second detection result comprises a person bounding box and the confidence of the person bounding box; and the business state of the store to be detected is determined according to the first detection result and the second detection result. The application can improve the accuracy of store business state detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a detection method and system for store business state, a terminal and a storage medium. BACKGROUND

[0002] With the continuous increase in the number of retail stores of the same brand, maintaining brand consistency and avoiding damage to consumer trust and loss of customers due to chaotic store operating hours have become key points for store compliance operation. The unified manager of the store will supervise the operating state of the store by means of network cameras installed in all stores under the brand and real-time monitoring of digital tasks.

[0003] In the prior art, the store door scene image of the physical store is usually collected as the basis for judging the operating state of the store. However, this method has very high requirements for the installation of camera equipment and is difficult to implement, which is extremely inconvenient for stores that have already installed equipment. In addition, relying solely on the store door scene to determine the opening and closing state of the physical store makes it difficult to completely determine whether the store is operating, and the detection method is too single and has a high misjudgment rate.

[0004] Therefore, how to improve the accuracy of store operating state detection is a technical problem to be solved by those skilled in the art. SUMMARY

[0005] To solve the above technical problems, the present application provides a detection method for store business state, which can improve the accuracy of store business state detection. The present application also provides a detection system for store business state, a terminal and a storage medium, which have the same technical effect.

[0006] The first object of the present application is to provide a detection method for store business state.

[0007] The above application object of the present application is achieved by the following technical solution:

[0008] A detection method for store business state, comprising:

[0009] obtaining a store scene image of a store to be tested;

[0010] obtaining a pre-trained business state detection model, wherein the business state detection model is improved based on a YOLO network framework and includes a backbone network, a neck network, a detection head and a state classification branch;

[0011] inputting the store scene image into the business state detection model, processing the store scene image via the backbone network and the neck network to obtain feature maps of at least three different scales, processing a feature map with the smallest resolution among the feature maps via the state classification branch to obtain a first detection result, and processing the feature maps of at least three different scales via the detection head to obtain a second detection result, wherein the first detection result includes probability distributions of four types of set states, and the set states include a light-on door-window state, a light-on no-door-window state, a light-off door-window state, and a light-off no-door-window state, and the second detection result includes a person bounding box and a confidence of the person bounding box;

[0012] determining a store business state of the store to be detected according to the first detection result and the second detection result.

[0013] Preferably, in the method for detecting the store business state, the state classification branch includes a global average pooling layer, three fully connected layers, and an output layer connected in sequence, and the processing of the feature map with the smallest resolution via the state classification branch to obtain the first detection result includes:

[0014] inputting the feature map with the smallest resolution into the global average pooling layer for processing to output a global feature vector, inputting the global feature vector into the first fully connected layer, and gradually reducing dimensions by the three fully connected layers, and introducing a LeakyReLU activation function between adjacent fully connected layers for calculation, and outputting a 4-dimensional feature vector from the third fully connected layer, inputting the 4-dimensional feature vector into the output layer for calculation via a Softmax activation function to output probability distributions of four types of set states.

[0015] Preferably, in the method for detecting the store business state, the determination of the store business state of the store to be detected according to the first detection result and the second detection result includes:

[0016] determining an output state according to the probability distributions of four types of set states in the first detection result, wherein the output state is one of a light-on door-window state, a light-on no-door-window state, a light-off door-window state, and a light-off no-door-window state;

[0017] according to the second detection result, eliminating the person bounding box with a confidence less than a preset confidence threshold, and determining a number of persons according to the remaining person bounding boxes.

[0018] According to the number of people and the output state, if the number of people is greater than or equal to a preset number of people threshold, and the output state is the light-on and no door and window state, it is determined that the to-be-tested store is in a business state; if the number of people is less than the preset number of people threshold, or the output state is any one of the light-off and door and window state and the light-off and no door and window state, it is determined that the to-be-tested store is in a non-business state.

[0019] Preferably, in the store business state detection method, the determining the output state according to the probability distribution of the four types of set states in the first detection result comprises:

[0020] According to the probability distribution of the four types of set states in the first detection result, the set state with the highest probability value is taken as the output state.

[0021] Preferably, in the store business state detection method, the determining the output state according to the probability distribution of the four types of set states in the first detection result comprises:

[0022] According to the first detection result, it is determined whether the probability value of the set state with the highest probability value is greater than or equal to a first preset threshold, and whether the difference between the probability value of the set state with the highest probability value and the probability value of the set state with the second highest probability value is greater than or equal to a second preset threshold, if so, the set state with the highest probability value is taken as the output state.

[0023] Preferably, in the store business state detection method, it further comprises:

[0024] If the probability value of the set state with the highest probability value is less than the first preset threshold, or the difference between the probability value of the set state with the highest probability value and the probability value of the set state with the second highest probability value is less than the second preset threshold, an uncertain state is taken as the output state.

[0025] The first two frames of store scene images of the store scene image are taken as associated store scene images, and the first detection result corresponding to the associated store scene images and the output state are obtained;

[0026] It is determined whether the output states corresponding to the associated store scene images are the same and whether the output states are the set states,

[0027] If so, the output state of the store scene image is adjusted from the uncertain state to the output state corresponding to the associated store scene image;

[0028] If not, according to the first detection result corresponding to the store scene image and the first detection result corresponding to the associated store scene image, weighted sum calculation is performed according to four types of set states, respectively, to obtain weighted probability values of four types of set states, and the output state of the store scene image is adjusted from an uncertain state to the set state with the highest weighted probability value.

[0029] Preferably, in the store business state detection method, the second detection result further includes probability distribution of two types of set personnel categories, and the set personnel categories include clerks and customers.

[0030] Preferably, in the store business state detection method, after the output state of the store scene image is adjusted from an uncertain state to the output state corresponding to the associated store scene image or the set state with the highest weighted probability value, the method further includes:

[0031] If the output state of the store scene image is light-off with doors and windows or light-off without doors and windows, according to the second detection result corresponding to the store scene image and the second detection result corresponding to the associated store scene image, the personnel bounding box with a confidence less than a preset confidence threshold is removed, the personnel bounding box with a higher probability value of the clerk category is taken as a clerk bounding box, and according to the number of the clerk bounding boxes and the confidence of the clerk bounding boxes of three frames of images, it is determined whether the number of clerks of three frames of images are all greater than or equal to a preset clerk number threshold and whether the average confidence of the clerk bounding boxes in three frames of images are all greater than or equal to a preset average confidence threshold, if yes, the output state of the store scene image is adjusted from light-off with doors and windows to light-on with doors and windows or from light-off without doors and windows to light-on without doors and windows.

[0032] If the output state of the store scene image is light-on with doors and windows or light-on without doors and windows, according to the second detection result corresponding to the store scene image and the second detection result corresponding to the associated store scene image, the personnel bounding box with a confidence less than a preset confidence threshold is removed, and according to the remaining personnel bounding boxes of three frames of images, it is determined whether the number of personnel of three frames of images are all zero and whether the collection time corresponding to three frames of images is in a non-business time, if yes, the output state of the store scene image is adjusted from light-on with doors and windows to light-off with doors and windows or from light-on without doors and windows to light-off without doors and windows.

[0033] Preferably, in the store business state detection method, the pre-trained business state detection model is obtained by:

[0034] obtain a store scene image sample carrying personnel target annotation information and state annotation information, wherein the personnel target annotation information is used to represent the bounding box position of each person in the store scene image and the set personnel category to which the person belongs, and the state annotation information is used to represent the set state category to which the store scene image belongs;

[0035] According to the store scene image sample, an initial business state detection model is trained to obtain a trained business state detection model.

[0036] Preferably, in the store business state detection method, the annotation rule of the state annotation information is:

[0037] If the image brightness of the store scene image is greater than or equal to a preset brightness threshold, and the occlusion rate of the door and window in the image is less than a first preset occlusion threshold, the set state category is annotated as open light with door and window.

[0038] If the image brightness of the store scene image is greater than or equal to a preset brightness threshold, and the occlusion rate of the door and window in the image is greater than or equal to a second preset occlusion threshold or the door and window are not within the image range, the set state category is annotated as open light without door and window.

[0039] If the image brightness of the store scene image is less than a preset brightness threshold, and the occlusion rate of the door and window in the image is less than a first preset occlusion threshold, the set state category is annotated as closed light with door and window.

[0040] If the image brightness of the store scene image is less than a preset brightness threshold, and the occlusion rate of the door and window in the image is greater than or equal to a second preset occlusion threshold or the door and window are not within the image range, the set state category is annotated as closed light without door and window.

[0041] Preferably, in the store business state detection method, the initial business state detection model is trained according to the store scene image sample to obtain a trained business state detection model, comprising:

[0042] According to the store scene image sample, the personnel detection loss and the state classification loss after weighted summation are used as a loss function to train the initial business state detection model until the model converges, and a trained business state detection model is obtained, wherein the expression of the loss function is:

[0043] Loss total =Loss detect +λ×Loss state ;

[0044] In the formula, Loss total indicates the loss function,Loss detect representing the person detection loss, Loss state representing the state classification loss, λ representing a weight coefficient.

[0045] Preferably, in the method for detecting the store business state, the person detection loss is obtained by summing a bounding box regression loss, a person category loss, and a confidence loss, wherein the bounding box regression loss adopts a CIoU Loss loss function, the person category loss adopts a binary Focal Loss loss function, and the confidence loss adopts a BCE Loss loss function.

[0046] Preferably, in the method for detecting the store business state, the state classification loss adopts a four-class weighted cross-entropy loss function.

[0047] Preferably, in the method for detecting the store business state, the weight coefficient λ has a dynamically set value, specifically:

[0048] If the current training round of the business state detection model belongs to a preset early training stage, the value of λ is determined as a first set value;

[0049] If the current training round of the business state detection model belongs to a preset middle training stage, the value of λ is adjusted to a second set value greater than the first set value;

[0050] If the current training round of the business state detection model belongs to a preset late training stage and the F1 score of the business state detection model on a validation set is less than a preset score threshold, the value of λ is adjusted to a third set value greater than the second set value.

[0051] Preferably, in the method for detecting the store business state, the YOLO network is specifically a YOLOv12 network.

[0052] A second object of the present application is to provide a system for detecting a store business state.

[0053] The above-mentioned second object of the present application is achieved by the following technical solution:

[0054] A system for detecting a store business state, comprising:

[0055] a first acquisition unit configured to acquire a store scene image of a store to be detected;

[0056] A second acquisition unit is configured to acquire a pre-trained business state detection model, wherein the business state detection model is improved based on a YOLO network framework and includes a backbone network, a neck network, a detection head, and a state classification branch.

[0057] A detection unit is configured to input a store scene image into the business state detection model, process the store scene image via the backbone network and the neck network to obtain at least three feature maps of different scales, process a feature map with the smallest resolution among the at least three feature maps via the state classification branch to obtain a first detection result, and process the at least three feature maps of different scales via the detection head to obtain a second detection result, wherein the first detection result includes a probability distribution of four set states, and the set states include a light-on door-window state, a light-on no door-window state, a light-off door-window state, and a light-off no door-window state, and the second detection result includes a person bounding box and a confidence of the person bounding box.

[0058] A determination unit is configured to determine a store business state of the to-be-detected store according to the first detection result and the second detection result.

[0059] A third object of the present application is to provide a detection terminal for a store business state.

[0060] The third object of the present application is achieved by the following technical solution.

[0061] A detection terminal for a store business state includes a storage medium and a processor.

[0062] The storage medium stores computer execution instructions.

[0063] The processor executes the computer execution instructions stored in the storage medium to implement the detection method for a store business state.

[0064] A fourth object of the present application is to provide a computer readable storage medium.

[0065] The fourth object of the present application is achieved by the following technical solution.

[0066] A computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by a processor to implement the detection method for a store business state.

[0067] The technical solution improves the existing YOLO network framework, adds a state classification branch on the basis of the existing YOLO network framework, and constructs a business state detection model with a double-task parallel architecture. Specifically, the backbone network and the neck network of the business state detection model are used to process the input store scene image, thereby obtaining at least three feature maps of different scales. The feature map containing global semantic information and having the smallest resolution is processed through the state classification branch to obtain a first detection result, thereby realizing the classification of four set states of "light on + door and window state"; the at least three feature maps of different scales are processed through the detection head to obtain a second detection result, thereby realizing the accurate detection of personnel in the store.

[0068] According to the first detection result and the second detection result, it can be determined whether there is personnel activity in the store and the light state of the store, so as to comprehensively determine the business state of the store, fuse the personnel and light and door and window double-core features, cover complex scenes, reduce the misjudgment rate based on single feature judgment, and solve the problems of single detection method and high misjudgment rate of the existing detection method.

[0069] In addition, the technical solution can adapt to different types of stores without relying on additional hardware devices, and has low deployment cost.

[0070] In summary, the technical solution can improve the accuracy of store business state detection. BRIEF DESCRIPTION OF DRAWINGS

[0071] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiment or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0072] Figure 1 It is a flowchart of a store business state detection method in an embodiment of the present application.

[0073] Figure 2 It is a structural diagram of a store business state detection system in an embodiment of the present application.

[0074] Figure 3 It is a structural diagram of a store business state detection terminal in an embodiment of the present application. DETAILED DESCRIPTION

[0075] In order to make the skilled in the art better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0076] In the embodiments provided in the present application, it should be understood that the disclosed methods and systems can be implemented in other ways. The system embodiments described below are only schematic. For example, the division of the modules is only a logical function division, and there can be another division in actual implementation. For example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the various components shown or discussed can be indirect coupling or communication connection through some interface, device or module, and can be electrical, mechanical or other forms.

[0077] It should be understood that if "system", "device", "unit" and / or "module" are used in the present application, it is only a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.

[0078] In addition, the terms "first", "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "multiple", "several" is two or more, unless otherwise explicitly and specifically limited.

[0079] If flowcharts are used in the present application, the flowcharts are used to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or subsequent operations are not necessarily executed in sequence. On the contrary, each step can be processed in reverse order or simultaneously. At the same time, other operations can be added to these processes, or one or more steps of the operation can be removed from these processes.

[0080] It is also to be noted that, in the present document, such as the terms "comprising", "including" or any other variant thereof, are intended to cover non-exclusive inclusions, such that an item or device that includes a list of elements is not limited to those elements, but can include other elements not expressly listed, or also include elements inherent to such item or device. Without more limitation, an element defined by the phrase "comprising a" does not exclude the presence of additional identical elements in the item or device including the element.

[0081] Embodiments of the present application are written in a progressive manner.

[0082] As shown in Figure 1 The embodiments of the present application provide a detection method of a store business state, including:

[0083] S101. Obtain a store scene image of a to-be-detected store;

[0084] In S101, specifically, the store scene image is obtained in real time through a camera deployed in the to-be-detected store, which can adapt to different types of stores, does not need to rely on additional hardware devices, and has low deployment cost. The collection frequency can be set to n frames / minute according to requirements, and the acquisition method of the store scene image can also be other methods, which is not limited in the present application.

[0085] S102. Obtain a pre-trained business state detection model, wherein the business state detection model is obtained based on improvement of a YOLO network framework, and includes a backbone network, a neck network, a detection head and a state classification branch;

[0086] In S102, specifically, YOLO (You Only Look Once) is a single-stage (one-stage) real-time target detection framework, which has developed multiple versions (such as YOLOv3, YOLOv4, YOLOv5, YOLOv8, etc.) since 2016. Although each version is different in details, the existing YOLO network usually adopts modular design, and the overall architecture can be divided into three core parts: backbone network, neck network and detection head. Among them, the backbone network is used to extract multi-scale features from the input image, the neck network is used to fuse the multi-scale features output by the backbone network, to enhance the semantic information and positioning ability of the features, and the detection head is used to directly predict the bounding box, confidence and / or category of the target based on the feature map output by the neck network.

[0087] In the embodiment, the existing YOLO network framework is improved, and a state classification branch is added on the basis of the existing YOLO network framework to construct a business state detection model with a double-task parallel architecture, including a backbone network, a neck network, a detection head, and the state classification branch. The business state detection model takes a store scene image as input, wherein the backbone network is used to process the store scene image to output multi-scale features, and the neck network is used to process the multi-scale features to output at least three features with different scales. The high-resolution feature map can capture small targets and local details in the image, the medium-resolution feature map can balance the details and global information to some extent, and the low-resolution feature map can capture large targets in the image and provide global context information. The backbone network and the neck network can adopt the network structure in the existing YOLO network framework, which is not limited in the present application.

[0088] In order to effectively determine the light state of the store and realize the precise classification function of "turning on / off the light + door and window state (image level)", the added state classification branch is used to process the feature map with the smallest resolution in the at least three features with different scales to obtain a first detection result, and the first detection result includes the probability distribution of four set states, including turning on the light with the door and window state, turning on the light without the door and window state, turning off the light with the door and window state, and turning off the light without the door and window state. "Illumination + door and window state" is an image-level global attribute (related to overall brightness and scene structure), and the door and window belong to the scene structure feature (whether visible / exists), which needs to be jointly judged with the illumination state (brightness). The feature map with the smallest resolution contains global semantic information, such as the semantic information of the door and window outline (such as the door frame and glass area), which can effectively assist the state classification branch to distinguish "with door and window" and "without door and window". The state classification branch can adopt the existing branch network structure for image classification, which is not limited in the embodiment.

[0089] In order to effectively determine whether there is personnel activity in the store and realize the precise detection of personnel (target level) in the store, the detection head processes the at least three features with different scales to obtain a second detection result, and the second detection result includes a personnel bounding box and a confidence of the personnel bounding box. Personnel detection relies on at least three resolution features with different scales to capture personnel local details (such as clothing and posture). The detection head can only perform a single-class detection task, i.e., detecting personnel in the store scene image. Based on the personnel bounding box, the personnel position can be located, and based on the confidence of the personnel bounding box, it can be determined whether it is personnel to filter background false detection. The detection head can adopt the network structure in the existing YOLO network framework, which is not limited in the embodiment.

[0090] After the initial business status detection model is constructed, a large number of store indoor scene images under different time periods, different passenger flows and different lighting conditions of the store can be collected, and the "personnel" human body target is labeled, and the light and door and window state of the image, that is, the above-mentioned set state category, is labeled to obtain pre-labeled store scene image samples, and the sample quantity is greater than or equal to a preset number, such as 5000. Then, the store scene image samples are used to train the initial business status detection model to obtain a trained business status detection model. The model training can be implemented based on an existing model training method, and the present embodiment is not limited in this regard.

[0091] In some embodiments, the at least three feature maps of different scales include a first feature map, a second feature map and a third feature map, wherein the resolution of the first feature map is greater than the resolution of the second feature map, and the resolution of the second feature map is greater than the resolution of the third feature map. Correspondingly, the state classification branch is configured to process the third feature map to obtain a first detection result, and the detection head is configured to process the first feature map, the second feature map and the third feature map to obtain a second detection result. For example, the resolution of the first feature map is 80x80, the resolution of the second feature map is 40x40, and the resolution of the third feature map is 20x20. The personnel detection relies on all resolution feature maps (20x20, 40x40, 80x80) output by the neck network to capture local details of the personnel, and the "light + door and window state" detection relies on a low-resolution feature map (20x20) output by the neck network, which contains global semantic information, but the present application is not limited thereto.

[0092] In other embodiments, the YOLO network is specifically a YOLOv12 network. As the first YOLO framework with attention mechanism as the core, YOLOv12 realizes a breakthrough balance between precision and speed through regional attention and FlashAttention technology, significantly improves the detection accuracy while maintaining real-time inference speed. In addition, the YOLOv12 model supports GPU acceleration, and the single-frame image detection time is less than or equal to 50 ms, which has strong real-time performance and can better meet the real-time monitoring needs of the store, but the present application is not limited thereto.

[0093] S103. inputting the store scene image into the business status detection model to obtain a first detection result and a second detection result;

[0094] In S103, specifically, after obtaining the pre-trained business status detection model, the store scene image is input into the business status detection model, and the store scene image is processed by the backbone network and the neck network to obtain at least three feature maps of different scales. The feature map with the smallest resolution is processed by the state classification branch to obtain a first detection result, and the at least three feature maps of different scales are processed by the detection head to obtain a second detection result.

[0095] Specifically, the input store scene image is processed by means of the backbone network and the neck network of the business status detection model, so as to obtain at least three feature maps of different scales. The feature map containing global semantic information and having the smallest resolution is processed by a state classification branch to obtain a first detection result, so as to realize 4-class setting state classification of "light on + door and window state", wherein the first detection result includes a probability distribution of 4-class setting states, and the setting states include light-on door and window state, light-on no door and window state, light-off door and window state, and light-off no door and window state. The at least three feature maps of different scales are processed by a detection head to obtain a second detection result, so as to realize accurate detection of personnel in the store, wherein the second detection result includes a personnel bounding box and a confidence of the personnel bounding box.

[0096] S104. According to the first detection result and the second detection result, the business status of the to-be-detected store is determined.

[0097] In S104, specifically, according to the probability distribution of the setting states in the first detection result, the light state of the to-be-detected store can be determined, and according to the personnel bounding box and the confidence of the personnel bounding box in the second detection result, whether there is personnel activity in the to-be-detected store can be determined. By comprehensively considering the two results, the business status of the to-be-detected store, such as the business status or the closed status, can be determined, so as to reduce the misjudgment rate based on a single feature and solve the problems of single detection method and high misjudgment rate in the prior art.

[0098] In some embodiments, one of the implementation manners of the present step specifically includes:

[0099] S1041. According to the probability distribution of the 4-class setting states in the first detection result, an output state is determined, wherein the output state is one of the light-on door and window state, the light-on no door and window state, the light-off door and window state, and the light-off no door and window state.

[0100] In S1041, specifically, according to the probability distribution of the four types of set states in the first detection result, the output state is determined to be one of the open light with door and window state, the open light without door and window state, the off light with door and window state, and the off light without door and window state. In some embodiments, one implementation of the present step specifically includes: according to the probability distribution of the four types of set states in the first detection result, the set state with the highest probability value is taken as the output state. For example, the probability distribution of the four types of set states is (P(L1), P(L2), P(D1), P(D2)), where L1 represents the open light with door and window, L2 represents the open light without door and window, D1 represents the off light with door and window, and D2 represents the off light without door and window. P(L1) represents the probability value of the open light with door and window, P(L2) represents the probability value of the open light without door and window, P(D1) represents the probability value of the off light with door and window, and P(D2) represents the probability value of the off light without door and window. The set state with the highest probability value is taken as the output state.

[0101] S1042. According to the second detection result, the personnel bounding box with a confidence less than a preset confidence threshold is removed, and the number of personnel is determined according to the remaining personnel bounding boxes.

[0102] In S1042, specifically, the preset confidence threshold can be set based on actual needs. For example, the preset confidence threshold can be set to 0.5. The personnel bounding box with a confidence greater than or equal to 0.5 is regarded as a valid personnel, and the personnel bounding box is retained. The personnel bounding box with a confidence less than 0.5 is regarded as an invalid personnel, and the personnel bounding box is removed. The number of personnel can be determined according to the number of remaining personnel bounding boxes.

[0103] S1043. According to the number of personnel and the output state, if the number of personnel is greater than or equal to a preset number of personnel threshold, and the output state is the open light without door and window state, it is determined that the to-be-tested store is in the business state. If the number of personnel is less than the preset number of personnel threshold, or the output state is any one of the off light with door and window state and the off light without door and window state, it is determined that the to-be-tested store is in the closed state.

[0104] In S1043, specifically, the preset personnel quantity threshold can be set based on actual needs, for example, the preset personnel quantity threshold can be set to 1; the light-on and no door and window state can represent that the store is in a light-on state, removing the interference factors of light-on and door and window, and light-off and door and window and light-off and no door and window can represent that the store is in a light-off state. If it is judged that the personnel quantity is greater than or equal to 1, and the output state is the light-on and no door and window state, it can be determined that the store under test has personnel activity and the store is in a light-on state, and then it is determined that the store under test is in a business state; if it is judged that the personnel quantity is less than 1, or the output state is any one of the light-off and door and window state and the light-off and no door and window state, it can be determined that the store under test has no personnel activity or the store is in a light-off state, and then it is determined that the store under test is in a non-business state, but the present application is not limited thereto.

[0105] In some other embodiments, if the output state is light-on and door and window, it can also be directly judged whether the personnel quantity is greater than or equal to the preset personnel quantity threshold, if yes, it is determined that the store under test is in a business state, if no, it is determined that the store under test is in a non-business state, and the present application does not make specific limitations thereto.

[0106] In the prior art, the store door scene image of the entity store is usually collected as the basis for judging the business state of the store. However, this way has extremely high requirements for the installation of the camera device, and the implementation difficulty is relatively large, which is extremely inconvenient for the stores that have already installed the device. In addition, relying only on the store door scene to judge the opening and closing state of the entity store, it is difficult to completely determine whether the store is in business, the detection method is too single, and the misjudgment rate is relatively high.

[0107] The above embodiments improve the existing YOLO network framework, add a state classification branch on the basis of the existing YOLO network framework, and construct a business state detection model with a double-task parallel architecture. Specifically, the backbone network and the neck network of the business state detection model are used to process the input store scene image, thereby obtaining at least three feature maps of different scales. The feature map containing global semantic information and having the smallest resolution is processed through the state classification branch to obtain a first detection result, realizing the 4-class setting state classification of "light-on and light-off + door and window state"; the at least three feature maps of different scales are processed through the detection head to obtain a second detection result, realizing the accurate detection of personnel in the store. According to the first detection result and the second detection result, it can be effectively determined whether there is personnel activity in the store and the light state of the store, so as to comprehensively determine the business state of the store, fuse the double-core features of personnel and light and door and window, cover complex scenes, reduce the misjudgment rate based on single feature judgment, solve the problem of single detection method and high misjudgment rate of the existing detection method. In addition, the above embodiments can adapt to different types of stores without relying on additional hardware devices, and the deployment cost is relatively low. In summary, the above embodiments can improve the accuracy of store business state detection.

[0108] In other embodiments of the present application, the state classification branch includes a global average pooling layer, three layers of fully connected layers (FC) and an output layer connected in sequence.

[0109] The global average pooling layer is configured to perform global average pooling (GAP) processing on the feature map with the minimum resolution, and output a global feature vector. Then, the global feature vector is input into the first layer of the fully connected layers, and is gradually reduced in dimension by the three layers of fully connected layers, and a LeakyReLU activation function is introduced between adjacent fully connected layers for calculation, and a four-dimensional feature vector is output by the third layer of the fully connected layers. The output layer is configured to calculate the probability distribution of four types of set states according to the four-dimensional feature vector by a Softmax activation function. Accordingly, one of the implementation manners of the step of processing the feature map with the minimum resolution through the state classification branch to obtain the first detection result includes: inputting the feature map with the minimum resolution into the global average pooling layer for processing to output a global feature vector, inputting the global feature vector into the first layer of the fully connected layers, and gradually reducing the dimension by the three layers of fully connected layers, and introducing a LeakyReLU activation function between adjacent fully connected layers for calculation, outputting a four-dimensional feature vector by the third layer of the fully connected layers, and inputting the four-dimensional feature vector into the output layer for calculation by the Softmax activation function to output the probability distribution of four types of set states.

[0110] Taking the third feature map with a resolution of 20x20 as an example, a 1x1xC global feature vector is extracted from the 20x20 third feature map by global average pooling, where C is the number of feature map channels; then the global feature vector is connected to the three layers of fully connected layers for gradual dimension reduction: 512→256→128→4, to obtain a four-dimensional feature vector, and the LeakyReLU activation function is used for intermediate calculation between adjacent fully connected layers to avoid gradient disappearance; finally, the output layer outputs the probability distribution of four types of states (P(L1), P(L2), P(D1), P(D2)) according to the four-dimensional feature vector by the Softmax activation function.

[0111] In this embodiment, the state classification branch includes a global average pooling layer, three layers of fully connected layers and an output layer connected in sequence, and the parameter scale is about 1 / 10 of the detection head, which avoids increasing too much calculation amount and increases the single-frame inference time consumption by ≤8 ms. While ensuring the detection accuracy, the complexity of the business state detection model is effectively controlled, so that the business state detection model detection process can stably output accurate detection results while maintaining high efficiency.

[0112] In other embodiments of the present application, the second detection result further includes a probability distribution of a 2-class set personnel category, and the set personnel category includes a store clerk and a customer.

[0113] Specifically, the detection head in the business status detection model processes at least three different scale feature maps to obtain a second detection result, and the second detection result includes a personnel bounding box, a confidence of the personnel bounding box, and a probability distribution of a 2-class set personnel category. That is, the detection head performs a multi-class detection task, and outputs “bounding box + class + confidence”. Here, the personnel category is expanded to two categories (“staff” and “customer”), the structure of the detection head remains unchanged, but the number of output channels is adjusted: each personnel bounding box corresponds to (4+1+2)-dimensional features (4 bounding box coordinates, 1 confidence, and 2 personnel category probabilities), 4 coordinates locate the position of the personnel; 1 confidence judges “whether it is a personnel” to filter background false detections; and 2 personnel category probabilities distinguish “whether it is a store clerk or a customer”.

[0114] In some embodiments, the business status detection model outputs “4+1+2”-dimensional features of all bounding boxes for each input store scene image. After filtering repeated boxes by non-maximum suppression (NMS), the final output format is as follows (using actual numerical values as an example):

[0115] [ # First bounding box: store clerk { "bbox": [120.5, 340.2, 80.3, 160.7], # bounding box (x1, y1, x2, y2, pixel coordinates) "confidence": 0.89, # confidence (probability of being a personnel) "class_probs": [0.92, 0.08], # personnel category probability: [store clerk probability, customer probability] "pred_class": "staff" # final predicted class (take the one with the maximum probability)}, # Second bounding box: customer { "bbox": [320.1, 280.5, 70.2, 150.3], "confidence": 0.85, "class_probs": [0.15, 0.85], "pred_class": "customer"} ]

[0116] Inference result analysis:

[0117] Bounding box (bbox): x1, y1 represents the top-left pixel coordinate of the box, x2, y2 represents the bottom-right pixel coordinate of the box, the position of the accurate boxer in the image can be used to count the distribution of the staff (such as whether the staff is at the cashier or whether the customer is in the shelf area). Confidence: used to filter low-confidence false detection boxes (such as considering boxes with confidence ≥ 0.5 as valid staff). Staff class probability (class_probs): reflects the possibility that the staff belongs to "staff" or "customer", for example, "[0.92, 0.08]" means 92% probability is staff and 8% is customer. Final prediction class (pred_class): determined by "taking the maximum value of staff class probability", directly output "staff" or "customer", which is convenient for subsequent business logic calls.

[0118] Correspondingly, in the step S1042, according to the second detection result, the bounding box of the staff with a confidence less than a pre-set confidence threshold is removed, and the number of staff and customers is determined respectively according to the remaining bounding box of the staff and the probability distribution of the two types of set staff classes, and the number of staff and customers is added to obtain the number of staff.

[0119] Specifically, first, valid staff screening is performed: only bounding boxes with "confidence ≥ 0.5" are retained, and low-confidence false detections (such as shelf shadows mistaken for staff) are removed. Then, the staff class probability calibration is performed: for the retained bounding boxes, the maximum value of the two types of probability (staff / customer) is taken as the final class (such as staff probability 0.65→judged as staff), and the probability value is recorded. Output format: a single frame of personnel detection result is simplified to {count_staff: number, count_customer: number, staff_probs: [list of staff probability], customer_probs: [list of customer probability]}. Finally, the number of staff and the number of customers are counted, and the number of staff and the number of customers are added to obtain the number of staff.

[0120] In other embodiments of the present application, another implementation of the above step S1041 specifically includes:

[0121] S201. According to the first detection result, it is judged whether the probability value of the set state with the highest probability value is greater than or equal to a first pre-set threshold value, and whether the difference between the probability value of the set state with the highest probability value and the probability value of the set state with the second highest probability value is greater than or equal to a second pre-set threshold value, if yes, execute S202, if no, execute S203;

[0122] S202. The set state with the highest probability value is taken as the output state;

[0123] S203. The uncertain state is taken as the output state;

[0124] In S201-S203, specifically, the first preset threshold and the second preset threshold can be set based on actual needs, for example, the first preset threshold can be set to 0.6, and the second preset threshold can be set to 0.2. Based on the probability distribution of the four types of set states in the first detection result, it is determined whether the probability value of the set state with the highest probability value among the four types of set states is greater than or equal to 0.6, and whether the difference between the highest probability value and the probability value of the set state with the second highest probability value is greater than or equal to 0.2. If yes, the set state corresponding to the highest probability value is taken as the output state. For example, the probability value P(L1) of the light-on with door and window L1 is 0.65>0.6, the probability value P(L2) of the light-on without door and window L2 is 0.3, which is the second highest probability value, and 0.65-0.3>0.2. It is confirmed that the output state is light-on with door and window, and then step S1042 can be further executed. The determination method in this step can avoid ambiguity caused by similar category probabilities, thereby improving the accuracy of subsequent store business state determination.

[0125] If the probability value of the set state with the highest probability value is less than the first preset threshold, or the difference between the probability value of the set state with the highest probability value and the probability value of the set state with the second highest probability value is less than the second preset threshold, the uncertain state is taken as the output state; for example, the probability value P(L1) of the light-on with door and window L1 is 0.52<0.6, the probability value P(L2) of the light-on without door and window L2 is 0.48, which is the second highest probability value, and 0.52-0.48<0.2. The uncertain state is taken as the output state, and the output format is: single frame state result is {status: "L1" / "L2" / "D1" / "D2" / "uncertain", probs: [L1_prob, L2_prob, D1_prob, D2_prob]}. "uncertain" represents the uncertain state, and step S204 is continued.

[0126] S204. The first two frames of the store scene images are taken as the associated store scene images, and the first detection result corresponding to the associated store scene images and the output state are obtained;

[0127] In S204, specifically, the first two frames of the store scene images of the current frame are taken as the associated store scene images, and then the two frames of the associated store scene images are respectively input into the above business state detection model to obtain the respective first detection result. Based on the respective first detection result, the respective output state is determined according to S201-S203.

[0128] In some embodiments, the ordered storage and calling of the associated store scene image can be realized by a sliding window cache. For example, the window size of the sliding window is fixed as 3 frames (current frame t, previous frame t-1, and previous two frames t-2), which ensures the coverage of state changes in a short time. The store state usually does not fluctuate dramatically within 3 frames. Assuming that the frame rate is 2 frames per second, 3 frames correspond to 1.5 seconds, which conforms to the actual scene. The cache update mechanism adopts "first in, first out (FIFO)", and the queue stores the processing results of the latest 3 frames (including the first detection result + the second detection result). Every time a new frame is input, the earliest frame is removed, and the window size is maintained unchanged. Boundary processing is performed. In the initial stage (less than 3 frames), if the current frame is in an "uncertain state" and there are only 1-2 frames in the cache, the result is temporarily stored and waits for 3 frames to be completed before being judged and processed. If there is an abnormal frame loss (such as camera lag), the previous frame result is used to fill in, and at most 1 frame is filled to avoid long-term distortion.

[0129] In addition, timing alignment verification can also be performed to ensure that the 3 frames in the cache belong to the same monitoring scene, and to avoid timing disorder caused by camera switching or angle mutation. The personnel position IOU (Intersection over Union, overlap) between consecutive frames is calculated, such as the main personnel bounding box IOU of t frame and t-1 frame ≥ 0.7, or the global brightness difference (brightness mean value change ≤ 10%). If it does not meet the condition, it is determined that the scene has mutated, and the cache is emptied and re-accumulated. The present application is not limited thereto.

[0130] S205. determining whether the output states corresponding to the associated store scene images are the same and whether the output states are the set states, if yes, adjusting the output state of the store scene image from the uncertain state to the output state corresponding to the associated store scene image; if no, performing weighted sum calculation according to the first detection result corresponding to the store scene image and the first detection result corresponding to the associated store scene image, respectively, according to the 4 types of set states, to obtain the weighted probability values of the 4 types of set states, and adjusting the output state of the store scene image from the uncertain state to the set state with the highest weighted probability value.

[0131] In S205, specifically, the voting statistics method is adopted, and the "uncertain state" image frame does not participate in voting and is only used as a reference. For example, the output state corresponding to the store scene image of the current frame t is the uncertain state, and does not participate in voting. If the output states corresponding to the associated store scene images of 2 frames are the same and are one of the 4 types of set states, such as the output states corresponding to the associated store scene images of the previous frame t-1 and the previous two frames t-2 are both open light with doors and windows L1, then the output state corresponding to the store scene image of the current frame t is adjusted from the uncertain state to the open light with doors and windows L1 corresponding to the associated store scene image, and then step S1042 can be further executed.

[0132] If the output states corresponding to the two frames of associated store scene images are different, or there is an uncertain state in the output states corresponding to the two frames of associated store scene images, such as the output state of the associated store scene image of the previous frame t-1 is open light with door and window L1, and the output state corresponding to the two frames t-2 of the associated store scene image is open light without door and window L2, then the weighted probability value calculation of the four types of set states of the subsequent three frames of store scene images is performed. Specifically, according to the probability distribution of the four types of set states in the first detection result corresponding to the three frames of store scene images, the weighted calculation is respectively performed according to the preset weight value (for example, the current frame t: weight 0.4, the latest frame information is the most reliable; the previous frame t-1: weight 0.3; the previous two frames t-2: weight 0.3) and the set state category, to obtain the weighted probability value of the four types of set states, and the calculation formula is: the weighted probability value of the set state S = (the probability of the set state S of the store scene image of the current frame t x 0.4) + (the probability of the set state S of the associated store scene image of the previous frame t-1 x 0.3) + (the probability of the set state S of the associated store scene image of the previous two frames t-2 x 0.3); wherein the set state S belongs to one of open light with door and window L1, open light without door and window L2, closed light with door and window D1, and closed light without door and window D2. Then the output state of the store scene image is adjusted from the uncertain state to the set state with the highest weighted probability value, and then step S1042 can be continuously executed. For example, the probabilities of the three frames of store scene images to open light with door and window L1 are 0.5 (t frame, uncertain), 0.6 (t-1 frame, determined), and 0.55 (t-2 frame, uncertain) → weighted probability value = 0.5 x 0.4 + 0.6 x 0.3 + 0.55 x 0.3 = 0.2 + 0.18 + 0.165 = 0.545; if the weighted probability value of open light without door and window L2 is 0.48, then the output state corresponding to the store scene image of the current frame t is determined to be open light with door and window L1.

[0133] Through the above steps, when a single frame image is determined to be an uncertain state, the detection results of the previous two frames of images are fused for comprehensive judgment, which can effectively reduce the misjudgment rate of the output state, and further improve the accuracy of the subsequent store business state judgment.

[0134] In other embodiments of the present application, after the above step S205, the method further comprises:

[0135] S206. If the output state of the store scene image is light-off with door and window or light-off without door and window, according to the second detection result corresponding to the store scene image and the second detection result corresponding to the associated store scene image, personnel bounding boxes with confidence less than a preset confidence threshold are removed, and a personnel bounding box with a higher probability value of a clerk category is taken as a clerk bounding box. According to the number of clerk bounding boxes of the three frames of images and the confidence of the clerk bounding boxes, it is determined whether the number of clerks of the three frames of images is all greater than or equal to a preset clerk number threshold, and whether the average confidence of the clerk bounding boxes in the three frames of images is all greater than or equal to a preset average confidence threshold. If yes, the output state of the store scene image is adjusted from light-off with door and window to light-on with door and window, or from light-off without door and window to light-on without door and window.

[0136] In S206, specifically, after the above comprehensive judgment of the detection results of the previous two frames of images, if the output state of the store scene image is light-off with door and window D1 or light-off without door and window D2, the correction judgment is continued. According to the second detection result corresponding to the store scene image of the current frame t, personnel bounding boxes with confidence less than a preset confidence threshold are removed to obtain remaining personnel bounding boxes. Then, a personnel bounding box with a higher probability value of a clerk category is taken as a clerk bounding box. According to the clerk bounding box, the number of clerks of the store scene image of the current frame t is determined, and the average confidence of the clerk bounding box is determined. In the same way, the number of clerks and the average confidence of the clerk bounding box corresponding to the associated store scene image of the previous frame t-1 and the associated store scene image of the previous two frames t-2 are also obtained. The preset clerk number threshold and the preset average confidence threshold can be set based on actual needs. For example, the preset clerk number threshold is 1, and the preset average confidence is 0.7. If the number of clerks of the three frames of images is all greater than or equal to 1, and the average confidence of the clerk bounding box in the three frames of images is all greater than or equal to 0.7, the correction is performed. The output state of the store scene image is adjusted from light-off with door and window D1 to light-on with door and window L1, or from light-off without door and window D2 to light-on without door and window L2.

[0137] S207. If the output state of the store scene image is light-on with door and window or light-on without door and window, according to the second detection result corresponding to the store scene image and the second detection result corresponding to the associated store scene image, personnel bounding boxes with confidence less than a preset confidence threshold are removed. According to the remaining personnel bounding boxes of the three frames of images, it is determined whether the number of personnel of the three frames of images is all zero, and whether the collection time corresponding to the three frames of images is in the non-business time. If yes, the output state of the store scene image is adjusted from light-on with door and window to light-off with door and window, or from light-on without door and window to light-off without door and window.

[0138] In S207, specifically, after the above-mentioned comprehensive judgment of the detection results of the fused two previous frames of images, if the output state of the store scene image is light on with door and window L1 or light on without door and window L2, the correction judgment is continued, the number of people in the current frame t, the previous frame t-1 and the previous two frames t-2 is determined respectively according to the above-mentioned number of people calculation method, and then it is judged whether the number of people in the three frames of images is all zero and whether the collection time corresponding to the three frames of images is in the non-business time (such as two o'clock in the morning). If yes, the correction is performed, the output state of the store scene image is adjusted from light on with door and window L1 to light off with door and window D1, or from light on without door and window L2 to light off without door and window D2.

[0139] In the embodiment, the decision of the above-mentioned step S205 is corrected in combination with the personnel detection results (number and probability of clerks / customers) of the three frames of images, the scene of "state and personnel contradiction" (such as the state tends to "light off" but a plurality of clerks are detected) is solved, the abnormal result of "state and actual business behavior contradiction" is further filtered through personnel feature cross verification, the misjudgment rate of the output state can be effectively reduced, and then the accuracy of the subsequent store business state judgment is improved.

[0140] In other embodiments of the present application, after the above-mentioned steps S206 or S207, the final state label (L1 / L2 / D1 / D2) can also be output, and the fusion confidence (0-1, such as 0.9 for three frames of consistent results, and 0.8 after correction) and the key basis (such as "2 frames of L1+1 frame of uncertainty, the highest weighted probability value, and 2 clerks are detected") can be attached. The present application is not limited thereto.

[0141] In other embodiments of the present application, if the output states of the current frame t, the previous frame t-1 and the previous two frames t-2 are all uncertain states, and the difference between the highest value and the second highest value of the weighted probability value is less than a third preset threshold value (such as 0.1), the sliding window can be extended to 5 frames (t-4 to t), and the above-mentioned fusion judgment process is repeated. If it still cannot be determined, it is judged in combination with the historical state (such as all being light on with door and window L1 for the previous 10 minutes, so the final output state is temporarily determined as light on with door and window L1). If the output states of the current frame t, the previous frame t-1 and the previous two frames t-2 frequently jump (light on with door and window L1→light off with door and window D1→light on with door and window L1 appear in 3 frames): it is determined as an abnormal state (such as light flickering), an alarm is triggered (manual review is notified), and the latest stable state (such as light on with door and window L1 for the previous two frames before the jump, so the final output state is temporarily determined as light on with door and window L1) is output. The present application is not limited thereto.

[0142] In other embodiments of the present application, one of the implementation manners of the above-mentioned step 102 specifically includes:

[0143] S301. Obtain a store scene image sample carrying personnel target annotation information and state annotation information, wherein the personnel target annotation information is used to represent the bounding box position of each personnel in the store scene image and the set personnel category to which the personnel belongs, and the state annotation information is used to represent the set state category to which the store scene image belongs;

[0144] In S301, specifically, a large number of store indoor scene images under different time periods, different passenger flows and different illumination conditions can be collected; then personnel target annotation and state annotation are performed respectively to obtain the store scene image sample carrying the personnel target annotation information and the state annotation information.

[0145] In some embodiments, the annotation rule of the personnel target annotation information is as follows:

[0146] Personnel annotation: all personnel in each image are annotated with a bounding box (x_center, y_center, width, height) and classified with a category label (“staff” or “customer”).

[0147] Staff: wearing a work uniform, near the cash register / operation table, performing work behaviors such as arranging goods, etc.

[0148] Customer: non-working state, carrying a shopping bag, near the shelf, etc.

[0149] In some embodiments, the annotation rule of the state annotation information is as follows:

[0150] If the image brightness of the store scene image is greater than or equal to a preset brightness threshold, and the occlusion rate of the door and window in the image is less than a first preset occlusion threshold, the state category annotation is set to “light on with door and window”.

[0151] If the image brightness of the store scene image is greater than or equal to a preset brightness threshold, and the occlusion rate of the door and window in the image is greater than or equal to a second preset occlusion threshold or the door and window are not within the image range, the state category annotation is set to “light off without door and window”.

[0152] If the image brightness of the store scene image is less than a preset brightness threshold, and the occlusion rate of the door and window in the image is less than a first preset occlusion threshold, the state category annotation is set to “light off with door and window”.

[0153] If the image brightness of the store scene image is less than a preset brightness threshold, and the occlusion rate of the door and window in the image is greater than or equal to a second preset occlusion threshold or the door and window are not within the image range, the state category annotation is set to “light off without door and window”.

[0154] The preset brightness threshold, the first preset shielding threshold and the second preset shielding threshold can be set based on actual needs, for example, the first preset shielding threshold is 30%, and the second preset shielding threshold is 70%, but the application is not limited thereto. The image brightness and the shielding rate of the door and window in the image can be calculated by using an existing image algorithm, for example, the shielding rate of the door and window in the image can be calculated by using image geometry, and the application is not limited thereto. It should be noted that extreme cases such as a half-opened roller shutter door should be excluded during labeling, and can be classified as “with door and window”, and the visibility of the main body of the door and window is used as the criterion.

[0155] In some embodiments, the image brightness is calculated as follows: the effective area framed on the store scene image is used to calculate the RGB gray mean value and the HSV brightness mean value respectively, and the image brightness is calculated by weighted summation calculation based on the RGB gray mean value and the HSV brightness mean value.

[0156] Specifically, in the labeling process of the store business state detection, the core of the brightness auxiliary judgment is to quantize the brightness feature of the image by using an algorithm, to provide an objective brightness reference threshold for the labeling personnel, to replace the subjective visual judgment, and to ensure the consistency of the “light on / light off” labeling. The following are specific implementation details:

[0157] First, the labeling personnel need to frame the effective area region_area of the store interior, the pixel coordinate range (x1, y1, x2, y2), and the pixel matrix img_inner=img[y1:y2, x1:x2] of the area is cropped by using OpenCV, and the brightness is calculated only for the sub-matrix; then, the image brightness is quantitatively calculated: the double-index of the RGB gray value mean+HSV brightness channel mean is used to calculate the brightness of the cropped pixel matrix area, the human eye perception and the algorithm stability are considered, and the calculation formula is as follows:

[0158] RGB gray mean value:

[0159] ;

[0160] HSV brightness mean value:

[0161] ;

[0162] The final image brightness is the weighted average of the double-index, balancing the stability and the perception consistency:

[0163] ;

[0164] In the formula, RGB gray mean value is denoted by HSV brightness mean value is denoted by image brightness is denoted by and denotes the set weight, preferably = 0.6, = 0.4, denotes the total number of pixels, denotes the gray value of the red channel of the i-th pixel point, denotes the gray value of the green channel of the i-th pixel point, denotes the gray value of the blue channel of the i-th pixel point, denotes the lightness value of the V channel of the i-th pixel point.

[0165] In this embodiment, the image brightness is calculated by weighted summation calculation according to the RGB gray mean value and the HSV lightness mean value. The V channel of HSV can effectively exclude the interference of colored objects (such as billboards, work clothes) on brightness judgment, is more accurate than single RGB gray, can replace subjective visual judgment, and ensures the consistency of "light on / light off" labeling.

[0166] In other embodiments, before the formal labeling starts, the preset brightness threshold of "light on / light off" can be determined through sample set calibration. The determination method of the preset brightness threshold is as follows:

[0167] Select a set number (such as 200) of "light on / light off" store scene image samples (100 each) that have been manually judged; calculate the image brightness of each image, and count the brightness distribution of the two types of samples; the optimal preset brightness threshold Tthreshold is automatically calculated by using the Otsu threshold segmentation algorithm, which can maximize the inter-class variance of "light on / light off" samples and ensure the discrimination.

[0168] In this embodiment, by calculating the preset brightness threshold Tthreshold , the labeling personnel can make a decision, and the preset brightness threshold Tthreshold and visual judgment can be combined to finally label the light on / off state.

[0169] In other embodiments, the personnel target labeling and the state labeling can also be combined into the same file. Only the identification rule of the image-level state label needs to be added on the basis of the original personnel target labeling format to ensure that the model can read the target-level and image-level information at the same time. The following is a specific combination scheme, which is an extension design compatible with YOLO format:

[0170] Format of the combined labeling file (.txt)

[0171] The file name corresponds to the image name (for example, the image store_001.jpg corresponds to the annotation store_001.txt), and the file content includes two parts: the first line is a fixed image-level state label (distinguished by a special identifier to avoid confusion with the person target); and the subsequent lines are person target annotations (the original YOLO format is followed).

[0172] txt format definition:

[0173] # The first line is a state label line (fixed format)

[0174] state: [state_id]

[0175] # The subsequent lines are person target lines (YOLO format)

[0176] [person_id] [x_center_norm] [y_center_norm] [width_norm] [height_norm]

[0177] [person_id] [x_center_norm] [y_center_norm] [width_norm] [height_norm]

[0178] Field description: state: [state_id]: image-level state label, state_id is 0-3 (corresponding to 4 states: 0=L1, 1=L2, 2=D1, 3=D2), with the state: prefix to clearly identify and distinguish from the person target line; [person_id]: person class ID (0=store clerk, 1=customer), consistent with the original format; coordinate values: all normalized to 0-1 values (retain 6 decimal places).

[0179] With the above annotation format design, the classic YOLO target detection format can be retained, and only the first line special identifier is extended to state label, without the need to modify the data loading code significantly, and the compatibility is good. The state prefix clearly distinguishes the image-level label from the target-level label, has uniqueness, and can avoid the model from mistakenly identifying the state ID as the person ID; all annotation information is stored in a single file, which is convenient for data set management, and no additional.csv state file needs to be maintained.

[0180] S302. According to the store scene image sample, the initial business state detection model is trained to obtain a trained business state detection model.

[0181] In S302, specifically, after obtaining the store scene image sample, the store scene image sample can be divided into a training set, a verification set and a test set, and then the initial business status detection model is trained, verified and tested by using the training set, the verification set and the test set, and finally a trained business status detection model is obtained. The application is not limited to this.

[0182] In some embodiments, one implementation of the present step specifically includes: training the initial business status detection model according to the store scene image sample, taking the weighted sum of the personnel detection loss and the state classification loss as the loss function, until the model converges, and obtaining the trained business status detection model, wherein the expression of the loss function is:

[0183] Loss total =Loss detect +λ×Loss state ;

[0184] In the formula, Loss total loss function, Loss detect personnel detection loss, Loss state state classification loss, λ weight coefficient. The personnel detection loss is obtained by summing the bounding box regression loss, the personnel category loss and the confidence loss, and the expression of the personnel detection loss is:

[0185] Loss detect =Loss box +Loss cls_person +Loss conf ;

[0186] In the formula, Loss box bounding box regression loss, Loss cls_person personnel category loss, Loss conf confidence loss.

[0187] The bounding box regression loss Loss box The CIoU Loss loss function is used to improve the positioning accuracy of the occluded / small target personnel.

[0188] The personnel category loss Loss cls_personThe Focal Loss loss function is used for binary classification to solve the imbalance problem of staff / customer samples, and the expression is as follows:

[0189] ;

[0190] In the formula, represents the probability of the personnel category output by the model, c=1 for staff and c=2 for customers; the parameter represents the weight of the sample, and the parameter =0.7 (because the staff sample is small), and the parameter =0.3; represents the hyperparameter for adjusting the difficulty of the sample, and the parameter =2.

[0191] Confidence loss Loss conf The BCE Loss loss function is used to determine whether the bounding box is a person.

[0192] For a multi-classification task of 4 types of setting states, the state classification loss is Loss state The weighted cross-entropy loss function is used for four classifications to solve the sample imbalance (because the sample proportion of “light on with doors and windows” is small), and the specific expression is as follows:

[0193] ;

[0194] In the formula, k=1, 2, 3, 4, corresponding to 4 types of setting states; is a one-hot label (for example, “light on with doors and windows” corresponds to , = = =0); is the probability of the kth type of setting state output by the model; is the class weight, which is set in inverse proportion to the sample proportion (for example, if the sample proportion of a certain class is 20%, then =1 / 0.2=5, and the total is normalized).

[0195] Wherein, the value of the weight coefficient λ is dynamically set, and specifically:

[0196] If the current training round of the business state detection model belongs to the preset initial training period (such as 1-20 epochs), determine λ as the first setting value, for example, λ =0.4, which prioritizes personnel detection, especially the basic accuracy of staff / customer differentiation;

[0197] If the current training round of the business status detection model belongs to the preset middle training stage (such as 21-40 epochs), adjust λ to a second set value greater than the first set value, such as λ = 0.6, to strengthen the learning of the state classification task.

[0198] If the current training round of the business status detection model belongs to the preset late training stage (after 41 epochs), and the F1 score of the business status detection model on the validation set is less than the preset score threshold (such as F1 < 0.85), adjust λ to a third set value greater than the second set value, such as λ to 0.7.

[0199] In this embodiment, by adjusting the weight coefficient λ , the attention degree of the model to personnel detection and state classification can be flexibly regulated. In addition, by using loss functions such as CIoU Loss, Focal Loss and BCE Loss, boundary box regression, personnel category classification and confidence judgment are optimized respectively, which effectively improves the detection performance of the model in complex scenes. At the same time, by applying the weighted cross-entropy loss function of four classifications, the problem of sample imbalance is also solved, further improving the accuracy of state classification.

[0200] In other embodiments of the present application, the initial business status detection model can also be trained in stages. Specifically, in stage 1, the backbone network is frozen, only the detection head and the state classification branch are trained, the learning rate is 1e-4, the batch size is 16, and the training is performed for 20 epochs. The purpose is to let the classification branch quickly adapt to the global features and avoid interference of the backbone network. In stage 2, the fine-tuning is unfrozen, all layers of the backbone network are unfrozen, the learning rate is reduced to 5e-5, the batch size is 8, and the training is performed for 40-60 epochs. The cosine annealing learning rate decay is adopted to enhance the adaptability of the feature extraction network to the joint task. The present application is not limited thereto.

[0201] As shown in Figure 2 , in another embodiment of the present application, a detection system for the business status of a store is provided, which comprises:

[0202] A first acquisition unit 10 is configured to acquire a store scene image of a store to be detected.

[0203] A second acquisition unit 11 is configured to acquire a pre-trained business status detection model, wherein the business status detection model is improved based on a YOLO network framework and comprises a backbone network, a neck network, a detection head and a state classification branch.

[0204] The detection unit 12 is configured to input a store scene image into a business state detection model, process the store scene image via a backbone network and a neck network to obtain at least three feature maps of different scales, process a feature map with the smallest resolution among the at least three feature maps via a state classification branch to obtain a first detection result, and process the at least three feature maps via a detection head to obtain a second detection result. The first detection result includes a probability distribution of four set states, and the set states include a light-on door-window-open state, a light-on door-window-closed state, a light-off door-window-open state, and a light-off door-window-closed state. The second detection result includes a person bounding box and a confidence of the person bounding box.

[0205] The determination unit 13 is configured to determine a business state of a to-be-detected store according to the first detection result and the second detection result.

[0206] As shown in the following Figure 3 In another embodiment of the present application, a detection terminal for a store business state is provided, which includes a storage medium 20 and a processor 21.

[0207] The storage medium 20 stores computer-executable instructions.

[0208] The processor 21 executes the computer-executable instructions stored in the storage medium 20 to implement the detection method for a store business state as described above.

[0209] The processor 21 can include one or more processing cores. The processor 21 executes various functions and processes data by running or executing instructions, programs, code sets or instruction sets stored in the storage medium 20, and calling data stored in the storage medium 20. The processor 21 can be at least one of an application-specific integrated circuit, a digital signal processor, a digital signal processing device, a programmable logic device, a field programmable gate array, a central processing unit, a controller, a microcontroller and a microprocessor. It can be understood that, for different devices, the electronic devices used to implement the functions of the processor 21 described above can also be other devices.

[0210] The storage medium 20 can be used to store instructions, programs, codes, code sets or instruction sets. The storage medium 20 can include a storage program area and a storage data area. The storage program area can store instructions for implementing an operating system, instructions for at least one function, and instructions for implementing the detection method for a store business state, etc. The storage data area can store data involved in the detection method for a store business state, etc.

[0211] In another embodiment of the present application, a computer-readable storage medium is provided, and the computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are executed by a processor, the computer-executable instructions are used to implement the detection method for a store business state as described above.

[0212] The computer readable storage medium can be a U disk, a mobile hard disk, a read-only memory, a random access memory, an optical disk, or other medium capable of storing program codes.

[0213] The above description of disclosed embodiments enables one of ordinary skill in the art to make or use the application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for detecting the operational status of a store, characterized in that, include: Obtain store scene images of the store to be tested; A pre-trained business status detection model is obtained, wherein the business status detection model is based on the YOLO network framework and includes: a backbone network, a neck network, a detection head, and a state classification branch; The store scene image is input into the business status detection model. The store scene image is processed by the backbone network and the neck network to obtain feature maps of at least three different scales. The feature map with the smallest resolution is processed by the state classification branch to obtain a first detection result. The feature maps of at least three different scales are processed by the detection head to obtain a second detection result. The first detection result includes the probability distribution of four set states, which include the state with lights on and doors and windows open, the state with lights on and no doors and windows open, the state with lights off and doors and windows open, and the state with lights off and no doors and windows open. The second detection result includes the personnel bounding box and the confidence of the personnel bounding box. Based on the first detection result and the second detection result, the store operation status of the store to be tested is determined; The step of determining the store operation status of the store to be tested based on the first detection result and the second detection result includes: Based on the probability distribution of the four types of set states in the first detection result, the output state is determined, wherein the output state is one of the following: light on with doors and windows, light on without doors and windows, light off with doors and windows, and light off without doors and windows. Based on the second detection result, the personnel bounding boxes with confidence levels less than a preset confidence threshold are removed, and the number of personnel is determined based on the remaining personnel bounding boxes; The test store is determined to be in a business state if the number of people is greater than or equal to a preset threshold and the output state is "lights on, no doors or windows"; otherwise, the test store is in a closed state if the number of people is less than the preset threshold or the output state is either "lights off, doors and windows on" or "lights off, no doors or windows". The step of determining the output state based on the probability distribution of the four types of set states in the first detection result includes: Based on the first detection result, determine whether the probability value of the setting state with the highest probability value is greater than or equal to the first preset threshold, and whether the difference between the probability value of the setting state with the highest probability value and the probability value of the setting state with the second highest probability value is greater than or equal to the second preset threshold. If so, the setting state with the highest probability value is taken as the output state. If the probability value of the set state with the highest probability value is less than the first preset threshold, or if the difference between the probability value of the set state with the highest probability value and the probability value of the set state with the second highest probability value is less than the second preset threshold, then the uncertain state is taken as the output state. The first two frames of the store scene image are used as the associated store scene image, and the first detection result and output status corresponding to the associated store scene image are obtained. Determine whether the output states corresponding to the associated store scene images are the same and whether the output state is the set state. If so, the output state of the store scene image is adjusted from an uncertain state to the output state corresponding to the associated store scene image; If not, then based on the first detection result corresponding to the store scene image and the first detection result corresponding to the associated store scene image, weighted summation is performed according to the four types of set states to obtain the weighted probability values ​​of the four types of set states, and the output state of the store scene image is adjusted from an uncertain state to the set state with the highest weighted probability value.

2. The method as described in claim 1, characterized in that, The state classification branch includes a globally average pooling layer, three fully connected layers, and an output layer connected in sequence. The step of processing the feature map with the lowest resolution through the state classification branch to obtain a first detection result includes: The feature map with the minimum resolution is input into the global average pooling layer for processing, and a global feature vector is output. The global feature vector is then input into the first fully connected layer, and the dimensionality is progressively reduced by the three fully connected layers. The LeakyReLU activation function is introduced between adjacent fully connected layers for calculation. The third fully connected layer outputs a 4-dimensional feature vector, which is then input into the output layer and calculated using the Softmax activation function to output the probability distribution of the four set states.

3. The method as described in claim 1, characterized in that, The second detection result also includes the probability distribution of two set personnel categories, namely, store clerks and customers.

4. The method as described in claim 3, characterized in that, After adjusting the output state of the store scene image from an uncertain state to the output state corresponding to the associated store scene image, or after adjusting the output state of the store scene image from an uncertain state to the set state with the highest weighted probability value, the method further includes: If the output state of the store scene image is "lights off with doors and windows" or "lights off without doors and windows", then based on the second detection result corresponding to the store scene image and the second detection result corresponding to the associated store scene image, the bounding boxes of personnel with confidence scores less than a preset confidence threshold are removed, and the bounding boxes of personnel with higher probability values ​​of the employee category are taken as employee bounding boxes. Based on the number of employee bounding boxes and the confidence scores of the employee bounding boxes in the three frames, it is determined whether the number of employees in the three frames is greater than or equal to a preset employee number threshold, and whether the average confidence scores of the employee bounding boxes in the three frames are greater than or equal to a preset average confidence threshold. If so, then the output state of the store scene image is adjusted from "lights off with doors and windows" to "lights on with doors and windows", or from "lights off without doors and windows" to "lights on without doors and windows". If the output state of the store scene image is "lights on with doors and windows" or "lights on without doors and windows", then based on the second detection result corresponding to the store scene image and the second detection result corresponding to the associated store scene image, the bounding boxes of people with a confidence level less than a preset confidence threshold are removed. Based on the remaining bounding boxes of people in the three frames, it is determined whether the number of people in the three frames is zero and whether the collection time corresponding to the three frames is during non-business hours. If so, then the output state of the store scene image is adjusted from "lights on with doors and windows" to "lights off with doors and windows", or from "lights on without doors and windows" to "lights off without doors and windows".

5. The method as described in claim 3, characterized in that, The process of obtaining a pre-trained business status detection model includes: Obtain store scene image samples carrying personnel target annotation information and status annotation information, wherein the personnel target annotation information is used to characterize the bounding box position of each person in the store scene image and the set personnel category to which they belong, and the status annotation information is used to characterize the set status category to which the store scene image belongs. Based on the store scene image samples, the initial business status detection model is trained to obtain a trained business status detection model.

6. The method as described in claim 5, characterized in that, The labeling rules for the status labeling information are as follows: If the brightness of the store scene image is greater than or equal to the preset brightness threshold, and the occlusion rate of doors and windows in the image is less than the first preset occlusion threshold, then the status category is set as "lights on, doors and windows present". If the brightness of the store scene image is greater than or equal to the preset brightness threshold, and the occlusion rate of the doors and windows in the image is greater than or equal to the second preset occlusion threshold or the doors and windows are not in the image range, then the status category is set as "lights on, no doors or windows". If the brightness of the store scene image is less than the preset brightness threshold, and the occlusion rate of doors and windows in the image is less than the first preset occlusion threshold, then the status category is set as "lights off, doors and windows present". If the brightness of the store scene image is less than the preset brightness threshold, and the occlusion rate of doors and windows in the image is greater than or equal to the second preset occlusion threshold, or if doors and windows are not within the image range, then the status category is set as "lights off, no doors or windows".

7. The method as described in claim 5, characterized in that, The step of training the initial business status detection model based on the store scene image samples to obtain the trained business status detection model includes: Based on the store scene image samples, the initial business status detection model is trained using a weighted summation of personnel detection loss and state classification loss as loss functions until the model converges, resulting in a trained business status detection model. The expression for the loss function is as follows: Loss total =Loss detect +λ×Loss state ； In the formula, Loss total Denotes the loss function, Loss detect This indicates the loss of personnel detection. Loss state This represents the state classification loss. λ This represents the weighting coefficient.

8. The method as described in claim 7, characterized in that, The personnel detection loss is obtained by summing the bounding box regression loss, the personnel category loss, and the confidence loss. The bounding box regression loss adopts the CIoU Loss loss function, the personnel category loss adopts the binary classification Focal Loss loss function, and the confidence loss adopts the BCE Loss loss function.

9. The method as described in claim 7, characterized in that, The state classification loss uses a four-class weighted cross-entropy loss function.

10. The method as described in claim 7, characterized in that, The weighting coefficient λ The value is set dynamically, specifically: If the current training round of the business status detection model belongs to the preset initial training phase, then determine λ Set the first value; If the current training round of the business status detection model is in the preset training middle stage, adjust... λ The second set value is greater than the first set value; If the current training epoch of the business status detection model belongs to the preset late training stage, and the F1 score of the business status detection model on the validation set is less than a preset score threshold, adjust... λ The third setting value is greater than the second setting value.

11. The method according to any one of claims 1 to 10, characterized in that, The YOLO network specifically refers to the YOLOv12 network.

12. A system for detecting the operational status of a store, characterized in that, The method for detecting the store's operating status as described in claim 1 includes: The first acquisition unit is used to acquire store scene images of the store to be tested; The second acquisition unit is used to acquire a pre-trained business status detection model, wherein the business status detection model is obtained by improving the YOLO network framework and includes: a backbone network, a neck network, a detection head, and a state classification branch. The detection unit is used to input the store scene image into the business status detection model. The store scene image is processed by the backbone network and the neck network to obtain feature maps of at least three different scales. The feature map with the smallest resolution is processed by the state classification branch to obtain a first detection result. The feature maps of at least three different scales are processed by the detection head to obtain a second detection result. The first detection result includes the probability distribution of four set states, which include the state with lights on and doors and windows, the state with lights on and no doors and windows, the state with lights off and doors and windows, and the state with lights off and no doors and windows. The second detection result includes personnel bounding boxes and the confidence of the personnel bounding boxes. The determining unit is used to determine the store operation status of the store to be tested based on the first detection result and the second detection result.

13. A terminal for detecting the operating status of a store, characterized in that, include: Storage media and processor; The storage medium stores computer-executed instructions. The processor executes computer execution instructions stored in the storage medium to implement the method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 11.

Images