Bill processing method, device, equipment and storage medium

Abstract

The application provides a bill processing method, device, equipment and storage medium, and the method comprises the steps of determining the motion trajectory of a user in a target space region; according to the motion trajectory of the user, when it is detected that the user enters a checkout region of the target space region, the goods consumed by the user in the target space region are acquired, and the consumption activity information corresponding to the target space region is acquired; and according to the goods consumed by the user and the consumption activity information, the consumption bill of the user is determined. That is, the application can effectively solve the difficulty that the intelligent shelf can only settle the goods of the shelf alone and cannot be combined with the consumption activity information corresponding to the target space region, thereby reducing the consumption cost of the user and improving the consumption experience of the user.

Description

Technical Field

[0001] This application relates to the field of smart retail technology, and in particular to a billing method, apparatus, device, and storage medium. Background Technology

[0002] With the continuous development of technology, smart retail has begun to permeate people's daily lives. The emergence of devices such as facial recognition payment and unmanned vending machines has enabled people to enjoy the convenience of self-service anytime and anywhere.

[0003] However, current unmanned vending machines cannot be combined with consumption activities (such as discounts for purchases over a certain amount), which leads to an increase in the cost of consumption for users. Summary of the Invention

[0004] This application provides a billing processing method, apparatus, device, and storage medium to reduce users' consumption costs.

[0005] Firstly, this application provides a billing processing method, including:

[0006] Determine the user's movement trajectory within the target spatial area;

[0007] Based on the user's movement trajectory, when the user is detected entering the checkout area of ​​the target space area, the system obtains the goods consumed by the user in the target space area and the consumption activity information corresponding to the target space area.

[0008] The user's consumption bill is determined based on the goods consumed by the user and the consumption activity information.

[0009] Secondly, a billing processing apparatus is provided, comprising:

[0010] The trajectory determination unit is used to determine the user's movement trajectory within the target spatial area;

[0011] The acquisition unit is configured to, based on the user's movement trajectory, acquire the goods consumed by the user in the target space area and acquire the consumption activity information corresponding to the target space area when the user is detected to have entered the checkout area of ​​the target space area;

[0012] The bill determination unit is used to determine the user's consumption bill based on the goods consumed by the user and the consumption activity information.

[0013] In some embodiments, the trajectory determination unit is specifically configured to acquire a first image of the user in the target space region captured by the camera at the current moment; perform target object detection on the first image to obtain N first detection boxes and first feature information of the first image, wherein the target object includes the user and N is a positive integer; and determine the motion trajectory of the user in the target space region at the current moment based on the N first detection boxes and the first feature information.

[0014] In some embodiments, the trajectory determination unit is specifically used to input the first image into the JDE model to obtain the N first detection boxes and the first feature information output by the JDE model.

[0015] Optionally, the JDE model may use convolutional layers to replace the Focus module.

[0016] Optionally, at least one activation function in the JDE model is the modified linear unit ReLU activation function.

[0017] Optionally, the JDE model is trained using a dataset of training image sequences containing human figures, and the training images are marked with human figure identifiers and bounding boxes.

[0018] In some embodiments, the trajectory determination unit is specifically used to acquire M first motion trajectories that already exist in the target space region at the current time, where M is a positive integer; match the M first motion trajectories with the N first detection boxes and the first feature information to obtain P second motion trajectories, where P is a positive integer; and determine the user's motion trajectory in the target space region at the current time from the P second motion trajectories.

[0019] In some embodiments, the trajectory determination unit is specifically used to match at least one track among the M first motion trajectories with the N first detection boxes based on the N first detection boxes and the first feature information, to obtain the P second motion trajectories.

[0020] In some embodiments, the trajectory determination unit is specifically configured to: perform cascade matching of Q1 first motion trajectories with the N first detection boxes and the first feature information to obtain a first matching result, wherein the Q1 first motion trajectories are the first motion trajectories matched in multiple consecutive frames among the M first motion trajectories, and Q1 is a positive integer; perform intersection-union (IOU) matching of the unmatched first motion trajectories and unmatched first detection boxes in the first matching result with Q2 first motion trajectories to obtain a second matching result, wherein the Q2 first motion trajectories are the first motion trajectories other than the Q1 first motion trajectories among the M first motion trajectories; and obtain P second motion trajectories based on the matched first motion trajectories in the first matching result, the matched first motion trajectories in the second matching result, and at least one of the unmatched first detection boxes in the second matching result.

[0021] In some embodiments, the trajectory determination unit is specifically configured to: obtain K1 high-confidence detection boxes and K2 low-confidence detection boxes from the N first detection boxes based on the confidence level of the first detection boxes, wherein the sum of K1 and K2 is less than or equal to N; for each of the K1 high-confidence detection boxes, match the high-confidence detection box with the M first motion trajectories to obtain a third matching result; for each of the K2 low-confidence detection boxes, match the low-confidence detection box with the first motion trajectory that was not matched in the third matching result to obtain a fourth matching result; and obtain P second motion trajectories based on the first motion trajectory matched in the third matching result, the first motion trajectory matched in the fourth matching result, and at least one of the high-confidence detection boxes that was not matched in the third matching result.

[0022] In some embodiments, the trajectory determination unit is specifically used to predict the j-th detection box of the j-th trajectory at the current time; match the area of ​​the i-th detection box with the area of ​​the j-th detection box to obtain a first matching value between the i-th detection box and the j-th track; and determine the target matching result between the i-th detection box and the j-th trajectory based on the first matching value between the i-th detection box and the j-th track; wherein, if the i-th detection box is the high-confidence detection box and the j-th track is a first track among the M first motion trajectories, then the target matching result is a third matching result; if the i-th detection box is the low-confidence detection box and the j-th track is a first motion trajectory that is not matched in the third matching result, then the target matching result is a fourth matching result.

[0023] In some embodiments, the trajectory determination unit is specifically configured to: determine a first feature value corresponding to the i-th detection box from the first feature information; extract a second feature value corresponding to the j-th detection box; determine the distance between the first feature value and the second feature value as a second matching value between the i-th detection box and the j-th track; and determine the target matching result between the i-th detection box and the j-th track based on the first matching value and the second matching value between the i-th detection box and the j-th track.

[0024] In some embodiments, the trajectory determination unit is specifically used to update the first motion trajectory matched in the third matching result and the first motion trajectory matched in the fourth matching result to obtain a second motion trajectory; and to create a new second motion trajectory for the high-confidence detection box that is not matched in the third matching result.

[0025] In some embodiments, the trajectory determination unit is specifically used to determine the second motion trajectory corresponding to the user's identifier among the P second motion trajectories as the user's motion trajectory within the target space area at the current moment.

[0026] In some embodiments, the bill determination unit is further configured to determine the time the user stayed at the shelf based on the user's movement trajectory; determine the user's consumption level for the goods on the shelf based on the time the user stayed at the shelf; and trace the source of suspicious bills based on the user's consumption level for the goods on the shelf and the user's movement trajectory.

[0027] Optionally, the longer a user stays at the shelf, the higher the user's spending level on the goods on the shelf.

[0028] Thirdly, an electronic device is provided, including a processor and a memory. The memory is used to store a computer program, and the processor is used to invoke and run the computer program stored in the memory to perform the methods of the first aspect or its implementations described above.

[0029] Fourthly, a chip is provided for implementing the methods of any one of the first to second aspects or their respective implementations. Specifically, the chip includes a processor for calling and running a computer program from a memory, causing a device on which the chip is installed to perform the methods of the first aspect or their respective implementations.

[0030] Fifthly, a computer-readable storage medium is provided for storing a computer program that causes a computer to perform the methods described in the first aspect or its various implementations.

[0031] In a sixth aspect, a computer program product is provided, including computer program instructions that cause a computer to perform the methods described in the first aspect or its various implementations.

[0032] In a seventh aspect, a computer program is provided that, when run on a computer, causes the computer to perform the methods described in the first aspect or its various implementations.

[0033] In summary, this application determines a user's movement trajectory within a target spatial area; based on this trajectory, when the user enters the checkout area of ​​the target spatial area, it acquires the goods consumed by the user within that area and the corresponding consumption activity information; and based on the consumed goods and the consumption activity information, it determines the user's bill. This application effectively solves the problem that smart shelves can only settle payments for goods on their own shelves and cannot be combined with consumption activity information corresponding to the target spatial area, thereby reducing user costs and improving the user experience. Furthermore, this application intelligently identifies the timing of payment after a user takes goods from different shelves through trajectory tracking, thus enabling the efficient use of multiple smart shelves to build smart unmanned supermarkets that support a wider range of operational strategies. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is a schematic diagram of an application scenario of this application;

[0036] Figure 2 A flowchart illustrating a billing method provided in an embodiment of this application;

[0037] Figure 3 This is a system architecture diagram related to the embodiments of this application;

[0038] Figure 4 This is a schematic diagram illustrating an application scenario according to an embodiment of this application;

[0039] Figure 5 A flowchart illustrating a billing processing method provided in an embodiment of this application;

[0040] Figure 6 This is a schematic diagram of the network structure of the JDE model according to an embodiment of this application;

[0041] Figure 7This is a schematic diagram of a process for determining a motion trajectory according to an embodiment of this application;

[0042] Figure 8 This is a schematic diagram of another process for determining a motion trajectory according to an embodiment of this application;

[0043] Figure 9 This is a schematic block diagram of a billing processing apparatus provided in an embodiment of this application;

[0044] Figure 10 This is a schematic block diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0045] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0046] It should be understood that, in the embodiments of the present invention, "B corresponding to A" means that B is associated with A. In one implementation, B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information.

[0047] In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0048] Furthermore, to facilitate a clear description of the technical solutions in the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and the terms "first" and "second" are not necessarily different.

[0049] To facilitate understanding of the embodiments of this application, the relevant concepts involved in the embodiments of this application will be briefly introduced as follows:

[0050] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0051] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0052] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0053] This application's embodiments relate to billing processing, for example, to self-service billing processing in unmanned vending areas.

[0054] With the continuous development of technology, smart retail has begun to permeate people's daily lives. The emergence of devices such as facial recognition payment and unmanned vending machines has enabled people to enjoy the convenience of self-service anytime and anywhere.

[0055] Figure 1 This is a schematic diagram of an application scenario of this application, such as Figure 1 As shown, the target space area can be a convenience store or supermarket, or a portion of a convenience store or supermarket. Multiple unmanned self-service shelves 110 (e.g., smart shelves) are placed in this target area, and the unmanned self-service shelves 110 are communicatively connected to the server 120. A user terminal 130 is equipped with a relevant user terminal, which is communicatively connected to the server 120.

[0056] The unmanned self-service shelf 110 contains goods. When user 140 takes an item from the unmanned self-service shelf 110, the unmanned self-service shelf 110 collects information about the item taken by user 140 and sends this information to server 120. Server 120 generates an invoice and sends it to the user terminal 130, which then deducts the corresponding amount from user 140's account. User 140 can view the invoice for the purchased goods through the user terminal.

[0057] In some embodiments, the user terminal 130 may be a smartphone, tablet computer, laptop computer, smart wearable device (e.g., smartwatch, smart helmet, smart glasses, etc.), etc.

[0058] Server 120 can be one or more. When there are multiple servers 120, at least two servers provide different services, and / or at least two servers provide the same service; this embodiment does not limit this. Server 120 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Server 120 can also be a node in a blockchain.

[0059] The user terminal 130 and the server 120 can be connected directly or indirectly through wired or wireless communication, and this application embodiment does not limit this.

[0060] The unmanned self-service shelf 110 and the server 120 can be directly or indirectly connected through wired or wireless communication, and this application embodiment does not limit this.

[0061] In some embodiments, the unmanned self-service shelf 110 can be an unmanned vending machine with a camera installed on the door frame. When a user opens the door to take goods, the camera on the door frame can capture an image of the goods taken by the user and send the captured image to the server 120. The server 120 recognizes the image of the goods, identifies the goods taken by the user, and generates a bill for the goods.

[0062] In some embodiments, the unmanned self-service shelf 110 can be an unmanned vending machine. Optionally, a goods collection device, such as a camera or sensor, can be installed at any location on the counter, around the counter, or at the bottom of the goods. When a user takes a goods, the goods collection device sends the collected information to the server 120, and the server 120 generates a bill for the goods.

[0063] Currently, in convenience stores and supermarkets, a single smart vending machine intelligently identifies and calculates the consumption of items on its own shelf. Payment is processed as soon as the user leaves the shelf, but it cannot be integrated with other products in the store for coordinated operations. Even if combined settlement is achieved by merging purchase orders from the same store within a certain period, this approach faces several challenges. Firstly, each new smart vending machine needs to be included in the calculation. Secondly, it's difficult to effectively assess the time interval for merging calculations. Too short an interval means payment might be made before the user leaves the store, while too long an interval might result in the user being counted as making a second purchase. Therefore, existing technological solutions cannot effectively address the intelligent operation problem of multiple shelves in a store. In other words, current unmanned vending machines cannot be integrated with promotional activities (such as discounts for purchases over a certain amount), leading to increased costs for users.

[0064] To address the aforementioned technical issues, this application proposes a billing method that utilizes individual detection and trajectory tracking algorithms to temporarily record purchased items as the user retrieves goods from each shelf. Only when the user enters the checkout area of ​​the target space is the purchase calculated and the bill deducted uniformly. This approach allows for indirect verification of smart shelf purchase records based on the user's trajectory, and enables the combination of items from multiple shelves to implement store promotions such as discounts for purchases exceeding a certain amount, thereby reducing the user's spending costs and improving the user experience.

[0065] The technical solutions of the embodiments of this application will be described in detail below through some examples. The following embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0066] Figure 2 This is a flowchart illustrating a billing processing method provided in an embodiment of this application. The executing entity in this embodiment is a device with billing processing functionality, such as a billing processing apparatus. Figure 1 The server in the middle.

[0067] like Figure 2 As shown, the method in this application embodiment includes:

[0068] S201. Determine the user's movement trajectory within the target spatial area.

[0069] S202. Based on the user's movement trajectory, when the user is detected entering the checkout area of ​​the target space area, obtain the goods consumed by the user in the target space area, and obtain the consumption activity information corresponding to the target space area.

[0070] S203. Determine the user's consumption bill based on the goods consumed and consumption activity information.

[0071] The billing method provided in this application involves the server tracking the user after detecting their entry into a target area, generating a real-time movement trajectory of the user within that area. Simultaneously, it records the items consumed by the user within the target area, such as items taken from shelves. Based on the user's movement trajectory, upon detecting the user's entry into the checkout area of ​​the target area, it retrieves the recorded items consumed and the corresponding consumption activity information for that area. Based on the items consumed and the consumption activity information (e.g., discounts), it generates the user's bill.

[0072] In some embodiments, after the server generates a user's consumption bill according to the method described above, it sends the bill to the user, for example, via the user's terminal. After the user confirms that the consumption bill is correct, the server deducts the corresponding amount.

[0073] In some embodiments, the server generates a user's expense bill, deducts the payment, and sends the expense bill and payment information to the user.

[0074] Merchants can set the target space area according to their needs in this application embodiment.

[0075] In one example, the target spatial area mentioned above can be the entire store area, such as a supermarket, convenience store, or shopping mall.

[0076] In one example, the target space area mentioned above can be a part of a store such as a supermarket, convenience store, or shopping mall. For example, the target space area is the vegetable section in a supermarket.

[0077] Merchants in this application embodiment can set the checkout area of ​​the target space region according to their actual needs.

[0078] In one example, the checkout area mentioned above could be the exit of the target space area.

[0079] In one example, the checkout area mentioned above can be a specific area within the target space, such as the area near the exit.

[0080] Figure 3 This is a system architecture diagram related to the embodiments of this application, such as... Figure 3 As shown, it includes: users, smart shelves, tracking devices, smart shelf back-end services, trajectory tracking and calculation services, store order calculation services, and store moving and living animal services.

[0081] Optionally, the aforementioned smart shelf backend services, trajectory tracking calculation services, store order calculation services, and store moving animal services can all be located on a single server.

[0082] Optionally, the aforementioned trajectory tracking calculation service and store order calculation service can be located on one server, while the smart shelf backend service and store moving animal service can be located on one or two other servers.

[0083] like Figure 3 As shown, user purchase information is collected from various smart shelves and transmitted to the smart shelf backend service for recording. User trajectory information is collected by one or more tracking devices and transmitted to the trajectory tracking and calculation service, which records when the user enters the store, which shelves they linger on, and when they leave. After leaving the store, the smart shelf backend service and the trajectory tracking and calculation service aggregate the information to the store order calculation service for unified consumption settlement. Simultaneously, the store order calculation service obtains consumption activity information from the store operations and activities service, processes it, and generates the user's consumption bill. Finally, the user's consumption bill is pushed to the user for notification.

[0084] This application embodiment does not limit the method for determining the user's movement trajectory within the target spatial area in S201 described above.

[0085] In one example, a specific chip worn by the user can be used to locate the user and determine their movement trajectory within a target spatial area. For instance, when a user enters the target spatial area, they need to carry a positioning card containing a positioning chip, which is connected to a backend server. This positioning card reports the user's location information within the target spatial area to the server in real time, and the server then generates the user's movement trajectory within the target spatial area based on the location information reported by the positioning card.

[0086] In this example, the above Figure 3 The tracking device in the image is the aforementioned positioning card.

[0087] In one example, a camera captures images of the user within a target spatial area, and the server identifies the images captured by the camera to generate the user's movement trajectory within the target spatial area.

[0088] In some examples, the user's motion trajectory within the target spatial area can also be generated based on other existing tracking methods, and this application embodiment does not limit this.

[0089] In some embodiments, the generated motion trajectory also records the time the user stayed at the corresponding shelf. Based on this, the embodiments of this application further include the following steps:

[0090] Step 1: Determine the time the user stays at the shelf based on the user's movement trajectory.

[0091] Step 2: Determine the user's consumption level for the goods on the shelf based on the time the user spends at the shelf.

[0092] In some embodiments, the longer a user stays at a shelf, the higher the user's consumption level for the goods on that shelf. For example, the ratio of the time a user stays at a shelf to a positive number is used to determine the user's consumption level for the goods on that shelf.

[0093] In some embodiments, different durations of stay correspond to different consumption levels. For example, the correspondence between stay duration and consumption level is shown in Table 1:

[0094] Table 1

[0095] Duration of stay Consumption level [a1, a2) A1 [a2, a3) A2 …… ….. [an-1, an) An-1

[0096] As shown in Table 1 above, for a certain shelf, assuming the user stays at the shelf for a time t2, and t2 is within the time period [a2, a3), it can be determined that the user's consumption level for the goods on that shelf is A2.

[0097] Step 3: Based on the user's spending level on the goods on the shelf and the user's movement trajectory, trace the source of suspicious bills.

[0098] For example, when bills can be traced, if a user has a high level of spending on goods on a shelf, but the user's movement trajectory does not appear in the checkout area of ​​that target space, the bill can be identified as a suspicious bill and traced.

[0099] This application records the user's spending levels on different shelves, using this data as evidence for the analysis of smart shelf consumption data and for reconciliation purposes.

[0100] For example, taking the method of the embodiments of this application as an example in an unmanned vending scenario, at the application layer, this application establishes a series of trajectory monitoring and behavior recording rules for unmanned vending scenarios.

[0101] In unmanned supermarkets, special areas should be clearly marked, such as... Figure 4 As shown, it includes a goods area and a checkout area, and an exit area can also be set up. If a user enters the store and then returns to the checkout area, it is considered that the transaction should be completed. The monitoring target is the user, and the curve trajectory is calculated by the algorithm to obtain the user's movement trajectory.

[0102] In addition to determining the user's checkout timing based on their movement trajectory, this application also calculates the time a user spends on a particular shelf to determine their spending level. The longer the stay, the higher the spending level for the goods on that shelf. This level can serve as an auxiliary analytical basis for tracing suspicious bills in unmanned supermarkets. For example, by querying different suspicious levels of a trajectory within a certain time period, the source of the incorrect bill can be quickly located, providing basic capability support for the construction of unmanned supermarkets. Through testing and experimental analysis, the method of this application achieves a trajectory tracking accuracy of over 95% in areas with good lighting conditions, and its accuracy in detecting intrusion and changes in spending levels has also been verified.

[0103] The billing method provided in this application determines the user's movement trajectory within a target spatial area. Based on the user's movement trajectory, when the user enters the checkout area of ​​the target spatial area, the method acquires the goods consumed by the user within the target spatial area and the corresponding consumption activity information. Based on the consumed goods and consumption activity information, the method determines the user's consumption bill. This application effectively solves the problem that smart shelves can only settle payments for goods on their own shelves and cannot be combined with consumption activity information corresponding to the target spatial area, thereby reducing user costs and improving the user experience. Furthermore, this application intelligently identifies the settlement time after a user takes goods from different shelves through trajectory tracking, thus enabling the efficient use of multiple smart shelves to build smart unmanned supermarkets that support a wider range of operational strategies.

[0104] The method for determining the user's movement trajectory within the target spatial region in S201 above will be described in detail below with reference to specific embodiments.

[0105] Figure 5 A flowchart illustrating a billing processing method provided in an embodiment of this application. The execution subject of this embodiment is the aforementioned... Figure 1 The server in the middle.

[0106] like Figure 5 As shown, the above S201 includes the following steps S501 to S503:

[0107] S501. Acquire the first image of the user in the target space area captured by the camera at the current moment.

[0108] The camera in this embodiment of the application captures a first image of the user in the target space area in real time.

[0109] S502. Detect the target object in the first image to obtain N first detection boxes and the first feature information of the first image.

[0110] This application does not limit the method of performing target detection on the first image in S502 above to obtain N first detection boxes and the first feature information of the first image.

[0111] In one example, a target detection model is used to detect targets in the first image, and the target detection result of the first image is obtained.

[0112] Optionally, the target object is the user, i.e., the human figure in the first image is inspected.

[0113] Optionally, the target object mentioned above includes users. That is, in addition to users, the target object to be detected may also include other objects. This application does not impose any restrictions on this.

[0114] In this example, there are no restrictions on the specific network structure of the target detection model. For example, the target detection model can be a graph neural network, an image convolutional neural network, an adversarial network, a self-written decoding network, etc.

[0115] The tracking scheme of the trajectory tracking calculation algorithm in this application can be divided into two parts: a target detection model with human detection parameters and a multi-target tracking algorithm structure. The target detection model can detect the bounding box of human targets and extract their appearance features from the input visual image, and then match the target with its motion trajectory based on the multi-target tracking algorithm. In this application, a system will be built in addition to the basic algorithm capability interface to monitor trajectories and shopping behavior.

[0116] The technical details of this application are divided into three layers: target detection model training, tracking algorithm logic layer, and behavior monitoring application layer.

[0117] In one possible implementation, the target detection model described above is JDE (Jointly learns the detector and embedding model).

[0118] At this time, S502 includes: inputting the first image into the JDE model to obtain N first detection boxes and first feature information output by the JDE model.

[0119] In some embodiments, the network structure of the JDE model of this application can be any existing structure. For example, the input end of the JDE model includes a Focus module for extracting feature information of different scales of the first image.

[0120] In some embodiments, to reduce the computational load of this application, the JDE model of this application uses convolutional layers to replace the computationally demanding Focus module. Furthermore, this application also implements the maximum suppression part of the detection results in the subsequent tracking algorithm logic layer.

[0121] In some embodiments, at least one activation function in the JDE model is the Modified Linear Unit (ReLU) activation function. That is, in this application embodiment, the ReLU activation function with lower computational cost is used instead of the more computationally intensive SiLU activation function.

[0122] In some embodiments, the JDE model described above is trained using a dataset of training image sequences containing human figures, and the training images are marked with human figure identifiers and bounding boxes, so that the trained JDE model can output the recognized human figure detection boxes.

[0123] Figure 6 This is a schematic diagram of the network structure of the JDE model involved in an embodiment of this application, as shown below. Figure 6 As shown, the JDE model consists of a convolutional layer (Conv), a CBR module, a CSP1-1 module, a CSP1-3 module, a CSP2-1 module, a spatial pyramid pooling (SPP) layer, and an upsampling unit (UpSample).

[0124] Figure 6 As shown, the CBR module consists of convolutional layers, batch normalization (BN) layers, and activation functions (Rulu). The CSP1-1 and CSP1-3 modules each consist of a CBR module, several residual units, convolutional layers, BN layers, and activation functions. The CSP2-1 module consists of several CBR modules, convolutional layers, BN layers, and activation functions.

[0125] like Figure 6 Assuming the first image is 1088x608x3, where 1088x608 represents the image's width and height, and 3 represents its RGB channels, then inputting the 1088x608x3 first image... Figure 6 The JDE model shown has an output consisting of three bounding box branches and one feature map branch. For example, Figure 6 In the first detection box branch, Output1, the output detection box information is 76x136x6. 76x136 can be understood as the size of the detection box, and 6 represents 6 different dimensions of parameters. Four parameters represent the position information of the detection box, one parameter represents the category of the target object in the detection box, and one parameter represents the confidence level of the target object in the detection box belonging to that category. For example... Figure 6 As shown, the detection box information output by the second detection box branch, Output2, is 38x68x6, where 38x68 represents the size of the detection box, and the meaning of 6 is the same as that of 6 in the detection box information output by the first detection box branch. Figure 6 As shown, the detection box information output by the third detection box branch OutPut3 is 19X34X6, where 19X34 is the size of the detection box, and the meaning of 6 is the same as above.

[0126] like Figure 6 As shown, in addition to outputting the detection box, the JDE model in this embodiment also outputs the first feature information of the first image. Optionally, the first feature information can be a feature map as shown in Table 5. The size of the first feature information can be set. Figure 6 The size of the first feature information shown is 76X136X512, which is just an example. This application includes, but is not limited to, this example.

[0127] It should be noted that, Figure 6 This application indicates an example of a JDE model involved in its embodiments. The JDE model in this application includes, but is not limited to, [other examples]. Figure 6 As shown, for example, it is possible to... Figure 6 The relevant modules can be deleted, added, or replaced.

[0128] After obtaining the N first detection boxes and the first feature information of the first image according to the above steps, the following step S503 is executed.

[0129] S503. Based on the N first detection boxes and the first feature information, determine the user's movement trajectory within the target space area at the current moment.

[0130] According to the embodiments of this application, a multi-target tracking method is used to determine the user's motion trajectory in the target space area at the current moment based on N first detection boxes and first feature information.

[0131] The specific implementation methods of the above S503 include, but are not limited to, the following:

[0132] Method 1 uses a motion trajectory generation model to predict the user's motion trajectory within the target space region at the current moment. Specifically, the N first detection boxes and first feature information detected at the current moment are input into the motion trajectory generation model to obtain the user's motion trajectory within the target space region at the current moment output by the motion trajectory generation model.

[0133] Method 2 involves using a trajectory matching algorithm to determine the user's movement trajectory within the target spatial region at the current moment based on N first detection boxes and first feature information. Specifically, S503 includes the following steps S503-A to S503-C:

[0134] S503-A: Obtain the M first motion trajectories that already exist in the target space region at the current moment, where M is a positive integer.

[0135] In some embodiments, the M first motion trajectories that already exist in the target space region at the current moment include the M first motion trajectories updated and generated at the previous moment and the saved unmatched trajectories.

[0136] S503-B: Based on N first detection boxes and first feature information, match M first motion trajectories with N first detection boxes to obtain P second motion trajectories, where P is a positive integer.

[0137] The implementation methods of the above S503-B include, but are not limited to, the following:

[0138] Method 1, the above S503-B includes the following S503-B1: Based on N first detection boxes and first feature information, at least one track of M first motion trajectories is matched with N first detection boxes to obtain P second motion trajectories.

[0139] In one implementation of Method 1, any one or more of the M first motion trajectories are matched one by one with the N foreign first detection boxes.

[0140] In another implementation of Method 1, S503-B1 includes the following steps S503-B11 and S503-B13:

[0141] S503-B11. Based on the N first detection boxes and the first feature information, the Q1 first motion trajectories are cascaded and matched with the N first detection boxes respectively to obtain the first matching result. The Q1 first motion trajectories are the first motion trajectories matched in multiple consecutive frames among the M first motion trajectories, and Q1 is a positive integer.

[0142] S503-B12. The first motion trajectory and the first detection box that are not matched in the first matching result are matched with Q2 first motion trajectories by Intersection Over Union (IOU) to obtain the second matching result. Q2 first motion trajectories are the first motion trajectories other than Q1 first motion trajectories among M first motion trajectories.

[0143] S503-B13. Based on the first motion trajectory matched in the first matching result, the first motion trajectory matched in the second matching result, and at least one of the first detection boxes that were not matched in the second matching result, P second motion trajectories are obtained.

[0144] Specifically, based on the number of matches, the aforementioned M first motion trajectories are classified. For example, the first motion trajectories that match in multiple consecutive frames (e.g., 3 frames) from the M first motion trajectories are obtained as Q1 first motion trajectories. The first motion trajectories other than Q1 from the M first motion trajectories are denoted as Q2 first motion trajectories. Based on N first detection boxes and first feature information, each first track in the Q1 first motion trajectories is cascaded and matched with each of the N first detection boxes to obtain a first matching result. This first matching result includes three types: track matched, track not matched, and detection box not matched.

[0145] In some embodiments, the steps in S503-B11 above, which involve cascading and matching Q1 first motion trajectories with N first detection boxes to obtain a first matching result, include the following steps: For the first motion trajectory 1 in the Q1 first motion trajectories, predict the second check box 1 of the first motion trajectory 1 at the current time, and extract the second feature information corresponding to the second detection box (for example, input the second detection box into a neural network to obtain the second feature information output by the neural network); For the first detection box 1 in the N first detection boxes, determine the second feature information corresponding to the first detection box 1 from the first feature information; Match the second feature information corresponding to the first detection box 1 with the second feature information corresponding to the second check box 1, for example, by calculating the distance between the two features to obtain the feature matching value 1.

[0146] Optionally, based on the aforementioned feature matching value 1, a first matching result between the first detection box 1 and the first motion trajectory 1 is determined.

[0147] Optionally, a motion matching value 1 is calculated between the first detection box 1 and the second detection box 1, for example, by calculating the motion matching value 1 between the first detection box 1 and the second detection box 1 using Mahalanobis distance. Then, a weighted sum of the feature matching value 1 and the motion matching value 1 is calculated, and a first matching result between the first detection box 1 and the first motion trajectory 1 is determined based on this weighted sum.

[0148] In some embodiments, the steps in S503-B11 above, which involve performing IOU matching between the unmatched first motion trajectory and the unmatched first detection box in the first matching result and Q2 first motion trajectories respectively, to obtain the second matching result, include the following steps: For the first motion trajectory 2 in the Q2 first motion trajectories, predict the second detection box 2 of the first motion trajectory 2 at the current time, calculate the IOU value between the first detection box 2 and the second detection box 2, specifically, calculate the ratio of the intersection and union of the first detection box 2 and the second detection box 2, that is, the overlap of the first detection box 2 and the second detection box 2, and determine the second matching result between the first detection box 2 and the first motion trajectory 2 based on the overlap.

[0149] The combination of the first and second matching results mentioned above may include three results: trajectory matching, trajectory not matching, and detection box not matching.

[0150] Next, based on the first and second matching results mentioned above, P second motion trajectories of the target spatial region at the current moment are generated.

[0151] In one embodiment, S503-B13 includes: updating the first motion trajectory matched in the first matching result and the first motion trajectory matched in the second matching result to obtain a second motion trajectory; and creating a new second motion trajectory for the first detection box that is not matched in the second matching result.

[0152] Optionally, for a first motion trajectory that is not matched in the first matching result and the second matching result, if the number of times the first motion trajectory is not matched is less than a preset value (e.g., 30 frames), the first motion trajectory is saved as one of the P second motion trajectories and the trajectory matching is performed in the next moment.

[0153] In some embodiments, the present application uses the DeepSORT (Deep Simple Online and Realtime Tracking) algorithm for multi-target tracking to perform trajectory tracking and generate P second motion trajectories.

[0154] For example, after the JDE module performs bounding box detection of human-shaped targets and extracts the first feature information of the first image, it inputs the detection box and the first feature information into the DeepSORT algorithm. Figure 7As shown, the DeepSORT algorithm is used to match detection boxes with the first motion trajectories. Specifically, for the confirmed Q1 first motion trajectories out of M first motion trajectories, a cascaded matching process is performed with N first detection boxes to obtain the first matching result. This first matching result includes three types of results: unmatched tracks, unmatched detections, and matched tracks. Figure 7 As shown, next, the unmatched first motion trajectories and unmatched first detection boxes in the first matching result are each matched with Q2 first motion trajectories using IOU to obtain the second matching result. This second matching result includes three types of results: unmatched tracks, unmatched detection boxes, and matched tracks. For unmatched first motion trajectories in the second matching result, if they are marked as 'unconfirmed' or the number of unmatched occurrences exceeds a preset maximum (max_age), the unmatched first motion trajectory is deleted. If the number of unmatched occurrences of the unmatched first motion trajectory is less than the preset maximum (max_age), the unmatched first motion trajectory is retained and recorded as the second motion trajectory. For unmatched detection boxes in the second matching result, if the confidence level of the detection box is high, a new trajectory is generated for the detection box, recorded as the second motion trajectory, but marked as 'unconfirmed' (because these targets may be noise output by the detector). If a match is found in three consecutive frames, the new trajectory is confirmed as valid and marked as 'confirmed'; otherwise, it is considered a noise trajectory and marked as 'deleted'. For the first motion trajectory that matches in the first and second matching results, the first detection box that matches is added to the trajectory, and then updated using a Kalman filter to obtain the second motion trajectory.

[0155] The above describes one implementation method for generating the second motion trajectory in S503-B. The following describes another implementation method for generating the second motion trajectory in S503-B.

[0156] Method 2, the above S503-B includes the following steps S503-B21 to S503-B23:

[0157] S503-B21. Based on the confidence level of the first detection box, obtain K1 high-confidence detection boxes and K2 low-confidence detection boxes from N first detection boxes, where the sum of K1 and K2 is less than or equal to N.

[0158] The target detected in this embodiment is a human figure. As described above, the information in the first detection frame includes the probability that the target within the first detection frame is a human figure, and this probability is recorded as the confidence level of the first detection frame. The higher the probability that the target within the first detection frame is a human figure, the higher the confidence level of the first detection frame.

[0159] Based on this, K1 high-confidence detection boxes and K2 low-confidence detection boxes can be selected from N first detection boxes according to their confidence levels, where the sum of K1 and K2 is less than or equal to N. Optionally, the aforementioned K1 high-confidence detection boxes are those detection boxes in the N first detection boxes whose confidence level is greater than a first threshold, and the K2 low-confidence detection boxes are those detection boxes in the N first detection boxes whose confidence level is less than a second threshold. Optionally, the first threshold and the second threshold are equal; alternatively, the first threshold is greater than the second threshold.

[0160] S503-B22. For each of the K1 high-confidence detection boxes, match the high-confidence detection box with M first motion trajectories to obtain the third matching result.

[0161] For example, each of the K1 high-confidence detection boxes is matched with each of the M first motion trajectories to obtain a third matching result.

[0162] In one example, a cascading matching method is used to match the high-confidence detection box with the first running trajectory to obtain a third matching result.

[0163] In another example, the IOU matching method is used to match the high-confidence detection box with the first running trajectory to obtain the third matching result.

[0164] In another example, cascading matching and IOU matching are used to match the high-confidence detection box with the first running trajectory to obtain the third matching result.

[0165] Optionally, other matching methods can be used to match the high-confidence detection box with the first running trajectory to obtain a third matching result. This step does not restrict this method.

[0166] S503-B23: For each of the K2 low-confidence detection boxes, match the low-confidence detection box with the first motion trajectory that was not matched in the third matching result to obtain the fourth matching result.

[0167] For example, each high-confidence detection box in the K2 low-confidence detection boxes is matched with each first motion trajectory in the first motion trajectory that did not match in the third matching result to obtain the fourth matching result.

[0168] In one example, a cascading matching method is used to match the low-confidence detection box with the first running trajectory to obtain the fourth matching result.

[0169] In another example, the IOU matching method is used to match the low-confidence detection box with the first running trajectory to obtain the fourth matching result.

[0170] In another example, cascaded matching and IOU matching are used to match the low-confidence detection box with the first running trajectory to obtain the fourth matching result.

[0171] Optionally, other matching methods can be used to match the low-confidence detection box with the first running trajectory to obtain a fourth matching result. This step does not restrict this method.

[0172] This tracking algorithm improves upon previous methods that directly discarded low-score bounding boxes and only matched and tracked high-score bounding boxes above a confidence threshold. It categorizes the N first detection boxes output by the JED model according to their confidence levels. First, high-confidence detection boxes are matched against M first motion trajectories. Then, low-confidence detection boxes are matched against motion trajectories not matched by high-confidence detection boxes (i.e., trajectories prone to interruption in the current frame). Motion trajectories that remain unmatched after these matching steps are retained. Unmatched high-confidence detection boxes create a new trajectory, while unmatched low-confidence detection boxes are discarded. Experiments show that this tracking method improves the completeness and continuity of trajectory tracking compared to the previous approach of directly discarding low-confidence detection boxes. For example, it can accurately track users even when their movements are incomplete in the first image.

[0173] In some embodiments, the method of matching the high-confidence detection box with M first motion trajectories in S503-B22 to obtain a third matching result may be different from the method of matching the low-confidence detection box with the first motion trajectories that did not match in the third matching result in S503-B23 to obtain a fourth matching result.

[0174] In some embodiments, the method of matching the high-confidence detection box with M first motion trajectories in S503-B22 to obtain a third matching result is the same as the method of matching the low-confidence detection box with the first motion trajectory that did not match in the third matching result in S503-B23 to obtain a fourth matching result.

[0175] The following describes another matching method in S503-B22 and / or S503-B23. For ease of description, the matching process between the i-th detection box and the j-th trajectory is used as an example. If the i-th detection box is a high-confidence detection box and the j-th trajectory is one of the first trajectories in the M first motion trajectories, then the following target matching result is the third matching result; if the i-th detection box is a low-confidence detection box and the j-th trajectory is a first motion trajectory that is not matched in the third matching result, then the following target matching result is the fourth matching result.

[0176] For example, the matching process between the i-th detection box and the j-th trajectory includes the following steps:

[0177] Step A1: Predict the j-th detection box of the j-th trajectory at the current time.

[0178] For example, the j-th trajectory is input into the neural network to predict the detection box at the current time. For ease of description, this predicted detection box is denoted as the j-th detection box.

[0179] For example, based on the running direction of the j-th trajectory, predict the detection box of the j-th trajectory at the current moment, and denot it as the j-th detection box.

[0180] Step A2: Match the area of ​​the i-th detection box with the area of ​​the j-th detection box to obtain the first matching value between the i-th detection box and the j-th track.

[0181] For example, the reciprocal of the ratio of the area of ​​the i-th detection box to the area of ​​the j-th detection box is determined as the first matching value between the i-th detection box and the j-th track.

[0182] For example, the reciprocal of the difference between the area of ​​the i-th detection box and the area of ​​the j-th detection box is determined as the first matching value between the i-th detection box and the j-th track.

[0183] For example, the area of ​​the i-th detection box is matched with the area of ​​the j-th detection box using IOU to obtain the first matching value between the i-th detection box and the j-th track.

[0184] Optionally, other methods can be used to match the area of ​​the i-th detection box with the area of ​​the j-th detection box to obtain the first matching value between the i-th detection box and the j-th track. This step does not restrict this method.

[0185] Step A3: Determine the target matching result between the i-th detection box and the j-th trajectory based on the first matching value between the i-th detection box and the j-th trajectory.

[0186] The implementation methods for step A3 above include, but are not limited to, the following:

[0187] The first implementation determines the target matching result between the i-th detection box and the j-th track based on the magnitude of the first matching value between the i-th detection box and the j-th track. For example, if the first matching value between the i-th detection box and the j-th track is greater than a threshold a, the i-th detection box is determined to match the j-th track; if the first matching value is less than a threshold a, the i-th detection box is determined not to match the j-th track; or, if the first matching value is less than a threshold b, the i-th detection box is determined not to match. Optionally, threshold b is less than threshold a.

[0188] At the tracking algorithm level, this application utilizes the first feature information provided by the JDE model to maintain the matching ability for previously encountered targets throughout the lifecycle when the detection algorithm fails to detect an object after it is occluded. Based on this, step A3 above also includes the following second implementation method.

[0189] The second implementation method, step A3 above, includes the following steps:

[0190] Step A31: Determine the first feature value corresponding to the i-th detection box from the first feature information.

[0191] For example, the center of the i-th detection box is determined, and the feature value corresponding to the center position of the i-th detection box in the first feature information is determined as the first feature value corresponding to the i-th detection box. The first feature value includes one feature.

[0192] For example, the first feature information is a feature map. The feature value corresponding to the i-th detection box region in the first feature information is determined as the first feature value corresponding to the i-th detection box. The first feature value includes multiple feature values.

[0193] Step A32: Extract the second feature value corresponding to the j-th detection box.

[0194] For example, the second feature value corresponding to the j-th detection box can be extracted using a deep network.

[0195] Optionally, the second eigenvalue is an eigenvector with the unit norm.

[0196] Optionally, the second feature value corresponding to the j-th detection box is the feature value of the detection box matched in the previous frame in the j-th trajectory.

[0197] Step A33: Determine the distance between the first feature value and the second feature value as the second matching value between the i-th detection box and the j-th track.

[0198] In one example, the cosine distance between the first feature value and the second feature value is determined as the second matching value between the i-th detection box and the j-th track.

[0199] In one example, the Mahalanobis distance between the first feature value and the second feature value is determined as the second matching value between the i-th detection box and the j-th track.

[0200] For example, the second matching value D between the i-th detection box and the j-th track is determined according to the following formula (2). M (x):

[0201]

[0202] Where x is the input parameter, for example, the first feature value of the i-th detection box, and μX is the expected (mean) value of x, for example, the second feature value corresponding to the j-th trajectory. Let x be the covariance matrix of the input x.

[0203] Optionally, when the above formula (2) is applied to determine the second matching value between N detection boxes and different trajectories, the above x = (x m ,x2,x3,x4,…,x n ) T ,μX=(μX1,μX2,μX3,μX4,…,μX n ) T .

[0204] Step A34: Determine the target matching result between the i-th detection box and the j-th trajectory based on the first matching value and the second matching value between the i-th detection box and the j-th trajectory.

[0205] For example, the target matching result between the i-th detection box and the j-th track is determined by weighting the first matching value and the second matching value between the i-th detection box and the j-th track.

[0206] For example, the arithmetic sum of the first and second matching values ​​between the i-th detection box and the j-th track is used to determine the target matching result between the i-th detection box and the j-th track.

[0207] Based on the above method, the third matching results between K1 high-confidence detection boxes and M first motion trajectories are determined, and the fourth matching results between K2 low-confidence detection boxes and the first motion trajectories that do not match in the third matching results are determined. Then, the following steps S503-B24 are executed.

[0208] S503-B24. Based on the first motion trajectory matched in the third matching result, the first motion trajectory matched in the fourth matching result, and at least one of the high-confidence detection boxes that were not matched in the third matching result, P second motion trajectories are obtained.

[0209] For example, the first motion trajectory matched in the third matching result and the first motion trajectory matched in the fourth matching result are updated to obtain the second motion trajectory; a new second motion trajectory is created for the high confidence detection box that is not matched in the third matching result.

[0210] In some embodiments, the process of determining the P second motion trajectories in the second method of S503-B described above is as follows: Figure 8 As shown, based on the confidence level of the first detection box, the server obtains K1 high-confidence detection boxes and K2 low-confidence detection boxes from N first detection boxes. The server loads the track, obtaining the M existing first motion trajectories at the current time. For each of the K1 high-confidence detection boxes, the server matches the high-confidence detection box with each of the M first motion trajectories, obtaining a third matching result. This third matching result includes three types of results: unmatched tracks, unmatched detections, and matched tracks. Figure 8 As shown, next, for each of the K2 low-confidence detection boxes, the low-confidence detection box is matched with the unmatched first motion trajectory in the third matching result to obtain the fourth matching result. This fourth matching result includes three results: unmatched tracks, unmatched detections, and matched tracks. Unmatched low-confidence detection boxes in the fourth matching result are directly deleted. For unmatched first motion trajectories in the fourth matching result, if they are marked as 'unconfirmed' or the number of unmatched occurrences is greater than the preset maximum number (max_age), the unmatched first motion trajectory is deleted. If the number of unmatched occurrences of the unmatched first motion trajectory is less than the preset maximum number (max_age), the unmatched first motion trajectory is retained and recorded as the second motion trajectory.

[0211] For high-confidence detection boxes that do not match in the third matching result, a new trajectory is generated for the high-confidence detection box, denoted as the second motion trajectory, but marked as 'unconfirmed' (because these targets may be noise output by the detector). If a match is obtained in 3 consecutive frames, the new trajectory is confirmed as valid and marked as 'confirmed'; otherwise, it is considered a noise trajectory and marked as 'deleted'. For the first motion trajectory that matches in the third and fourth matching results, the matched first detection box is added to the matched first motion trajectory, and updated using a Kalman filter to obtain the second motion trajectory.

[0212] Based on the above method, P second motion trajectories of the target space region at the current moment can be obtained.

[0213] S503-C: Determine the user's current motion trajectory within the target space area from the second motion trajectory P.

[0214] In this application, for multi-target tracking, the tracking algorithm should be able to assign a unique identifier (TrackID) to each person appearing and disappearing from the video frame. Regardless of whether the target is occluded, severely deformed, or too close to other targets (interfering with each other), as long as the target is correctly detected, the tracking algorithm should be able to correctly associate it. That is, each trajectory includes an identifier (e.g., a user identifier). Based on this identifier, the user's current motion trajectory within the target space region can be determined from P second motion trajectories. For example, the second motion trajectory corresponding to the user's identifier among the P second motion trajectories can be determined as the user's current motion trajectory within the target space region.

[0215] The billing method provided in this application embodiment acquires a first image of the user in a target space region captured by a camera at the current moment, performs target object detection on the first image to obtain N first detection boxes and first feature information of the first image, and determines the user's movement trajectory in the target space region at the current moment based on the N first detection boxes and the first feature information, thereby achieving accurate determination of the user's movement trajectory in the target space region. When generating the user's consumption bill based on the accurately determined movement trajectory, the accuracy of the consumption bill can be improved.

[0216] The preferred embodiments of this application have been described in detail above with reference to the accompanying drawings. However, this application is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this application, various simple modifications can be made to the technical solutions of this application, and these simple modifications all fall within the protection scope of this application. For example, the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, this application will not describe the various possible combinations separately. Furthermore, various different embodiments of this application can also be arbitrarily combined, as long as they do not violate the spirit of this application, they should also be considered as the content disclosed in this application.

[0217] It should also be understood that, in the various method embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0218] The above text combined Figures 2 to 8 The method embodiments of this application have been described in detail below, and the device embodiments of this application are described in detail below.

[0219] Figure 9 This is a schematic block diagram of a billing processing apparatus provided in an embodiment of this application.

[0220] like Figure 9 As shown, the bill processing device 10 may include:

[0221] The trajectory determination unit 11 is used to determine the user's motion trajectory within the target spatial area;

[0222] The acquisition unit 12 is used to acquire, based on the user's movement trajectory, the goods consumed by the user in the target space area when the user is detected to have entered the checkout area of ​​the target space area, and to acquire the consumption activity information corresponding to the target space area;

[0223] The bill determination unit 13 is used to determine the user's consumption bill based on the goods consumed by the user and the consumption activity information.

[0224] In some embodiments, the trajectory determination unit 11 is specifically used to acquire a first image of the user in the target space region captured by the camera at the current moment; to perform target object detection on the first image to obtain N first detection boxes and first feature information of the first image, wherein the target object includes the user and N is a positive integer; and to determine the motion trajectory of the user in the target space region at the current moment based on the N first detection boxes and the first feature information.

[0225] In some embodiments, the trajectory determination unit 11 is specifically used to input the first image into the JDE model to obtain the N first detection boxes and the first feature information output by the JDE model.

[0226] Optionally, the JDE model may use convolutional layers to replace the Focus module.

[0227] Optionally, at least one activation function in the JDE model is the modified linear unit ReLU activation function.

[0228] Optionally, the JDE model is trained using a dataset of training image sequences containing human figures, and the training images are marked with human figure identifiers and bounding boxes.

[0229] In some embodiments, the trajectory determination unit 11 is specifically used to obtain M first motion trajectories that already exist in the target space region at the current time, where M is a positive integer; match the M first motion trajectories with the N first detection boxes and the first feature information to obtain P second motion trajectories, where P is a positive integer; and determine the user's motion trajectory in the target space region at the current time from the P second motion trajectories.

[0230] In some embodiments, the trajectory determination unit 11 is specifically used to match at least one track among the M first motion trajectories with the N first detection boxes based on the N first detection boxes and the first feature information to obtain the P second motion trajectories.

[0231] In some embodiments, the trajectory determination unit 11 is specifically configured to: perform cascade matching of Q1 first motion trajectories with the N first detection boxes and the first feature information to obtain a first matching result, wherein the Q1 first motion trajectories are the first motion trajectories matched in multiple consecutive frames among the M first motion trajectories, and Q1 is a positive integer; perform intersection-union (IOU) matching of the unmatched first motion trajectories and unmatched first detection boxes in the first matching result with Q2 first motion trajectories to obtain a second matching result, wherein the Q2 first motion trajectories are the first motion trajectories other than the Q1 first motion trajectories among the M first motion trajectories; and obtain P second motion trajectories based on the matched first motion trajectories in the first matching result, the matched first motion trajectories in the second matching result, and at least one of the unmatched first detection boxes in the second matching result.

[0232] In some embodiments, the trajectory determination unit 11 is specifically configured to: obtain K1 high-confidence detection boxes and K2 low-confidence detection boxes from the N first detection boxes based on the confidence level of the first detection boxes, wherein the sum of K1 and K2 is less than or equal to N; for each of the K1 high-confidence detection boxes, match the high-confidence detection box with the M first motion trajectories to obtain a third matching result; for each of the K2 low-confidence detection boxes, match the low-confidence detection box with the first motion trajectory that was not matched in the third matching result to obtain a fourth matching result; and obtain P second motion trajectories based on the first motion trajectory matched in the third matching result, the first motion trajectory matched in the fourth matching result, and at least one of the high-confidence detection boxes that was not matched in the third matching result.

[0233] In some embodiments, the trajectory determination unit 11 is specifically used to predict the j-th detection box of the j-th trajectory at the current time; match the area of ​​the i-th detection box with the area of ​​the j-th detection box to obtain a first matching value between the i-th detection box and the j-th track; and determine the target matching result between the i-th detection box and the j-th trajectory based on the first matching value between the i-th detection box and the j-th track; wherein, if the i-th detection box is the high-confidence detection box and the j-th track is a first track among the M first motion trajectories, then the target matching result is a third matching result; if the i-th detection box is the low-confidence detection box and the j-th track is a first motion trajectory that is not matched in the third matching result, then the target matching result is a fourth matching result.

[0234] In some embodiments, the trajectory determination unit 11 is specifically configured to: determine a first feature value corresponding to the i-th detection box from the first feature information; extract a second feature value corresponding to the j-th detection box; determine the distance between the first feature value and the second feature value as a second matching value between the i-th detection box and the j-th track; and determine the target matching result between the i-th detection box and the j-th track based on the first matching value and the second matching value between the i-th detection box and the j-th track.

[0235] In some embodiments, the trajectory determination unit 11 is specifically used to update the first motion trajectory matched in the third matching result and the first motion trajectory matched in the fourth matching result to obtain a second motion trajectory; and to create a new second motion trajectory for the high confidence detection box that is not matched in the third matching result.

[0236] In some embodiments, the trajectory determination unit 11 is specifically used to determine the second motion trajectory corresponding to the user's identifier among the P second motion trajectories as the user's motion trajectory within the target space area at the current moment.

[0237] In some embodiments, the bill determination unit 13 is further configured to determine the time the user stayed at the shelf based on the user's movement trajectory; determine the user's consumption level for the goods on the shelf based on the time the user stayed at the shelf; and trace the source of suspicious bills based on the user's consumption level for the goods on the shelf and the user's movement trajectory.

[0238] Optionally, the longer a user stays at the shelf, the higher the user's spending level on the goods on the shelf.

[0239] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, further details will not be provided here. Specifically, Figure 9 The apparatus shown can perform the embodiments of the above-described methods, and the foregoing and other operations and / or functions of each module in the apparatus are respectively for implementing the method embodiments corresponding to the electronic device. For the sake of brevity, they will not be described in detail here.

[0240] The apparatus of this application embodiment has been described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that this functional module can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, the steps of the method embodiments in this application can be completed by integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the method disclosed in this application embodiment can be directly embodied as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. Optionally, the software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps in the above method embodiments.

[0241] Figure 10 This is a schematic block diagram of an electronic device provided in an embodiment of this application. The electronic device may be as described above. Figure 2 The server or terminal shown, or the control device in the cooling system. This electronic device is used to execute the above-described model training method embodiments and / or cooling system control method embodiments.

[0242] like Figure 10 As shown, the electronic device 30 may include:

[0243] The system includes a memory 31 and a processor 32. The memory 31 stores a computer program 33 and transfers the program code 33 to the processor 32. In other words, the processor 32 can retrieve and run the computer program 33 from the memory 31 to implement the methods described in the embodiments of this application.

[0244] For example, the processor 32 can be used to execute the above method steps according to the instructions in the computer program 33.

[0245] In some embodiments of this application, the processor 32 may include, but is not limited to:

[0246] General-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0247] In some embodiments of this application, the memory 31 includes, but is not limited to:

[0248] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0249] In some embodiments of this application, the computer program 33 may be divided into one or more modules, which are stored in the memory 31 and executed by the processor 32 to complete the page recording method provided in this application. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 33 in the electronic device.

[0250] like Figure 10 As shown, the electronic device 30 may further include:

[0251] Transceiver 34, which can be connected to processor 32 or memory 31.

[0252] The processor 32 can control the transceiver 34 to communicate with other devices; specifically, it can send information or data to other devices or receive information or data sent by other devices. The transceiver 34 may include a transmitter and a receiver. The transceiver 34 may further include antennas, and the number of antennas may be one or more.

[0253] It should be understood that the various components in the electronic device 30 are connected through a bus system, which includes a data bus, a power bus, a control bus, and a status signal bus.

[0254] According to one aspect of this application, a computer storage medium is provided that stores a computer program thereon, which, when executed by a computer, enables the computer to perform the methods of the above-described method embodiments. Alternatively, embodiments of this application also provide a computer program product containing instructions that, when executed by a computer, cause the computer to perform the methods of the above-described method embodiments.

[0255] According to another aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the method described in the above-described method embodiments.

[0256] In other words, when implemented using software, it can be implemented wholly or partially in the form of a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0257] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0258] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0259] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. For example, the functional modules in the various embodiments of this application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0260] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A billing processing method, characterized in that, include: Determine the user's movement trajectory within the target space area at the current moment, and based on the user's movement trajectory, when the user is detected entering the checkout area of ​​the target space area, obtain the goods consumed by the user within the target space area, and obtain the consumption activity information corresponding to the target space area; The user's consumption bill is determined based on the goods consumed by the user and the consumption activity information; Determining the user's movement trajectory within the target spatial region at the current moment includes: Obtain the location information reported by the positioning chip carried by the user, and generate the user's movement trajectory within the target space area at the current moment based on the location information reported by the positioning chip; or, The system acquires a first image of the user in the target space region captured by the camera at the current moment, and M existing first motion trajectories in the target space region at the current moment, where M is a positive integer; it performs target object detection on the first image to obtain N first detection boxes and first feature information of the first image, and matches the M first motion trajectories with the N first detection boxes based on the N first detection boxes and the first feature information to obtain P second motion trajectories, where the target object includes the user, and N and P are both positive integers; it then determines the user's motion trajectory in the target space region at the current moment from the P second motion trajectories. The step of matching the M first motion trajectories with the N first detection boxes and the first feature information to obtain P second motion trajectories includes: Predict the second detection box of each of the Q1 first motion trajectories at the current time, and extract the second feature information corresponding to each second detection box. Determine the second feature information corresponding to each of the N first detection boxes from the first feature information. Match the second feature information corresponding to each second detection box with the second feature information corresponding to each first detection box to obtain a first matching result. The Q1 first motion trajectories are the first motion trajectories matched in multiple consecutive frames among the M first motion trajectories. The second feature information corresponding to the first detection box is the feature value corresponding to the center position of the first detection box in the first feature information. The second feature information corresponding to the second detection box is the second feature information corresponding to the first detection box matched by the first motion trajectory in the previous frame. The first motion trajectory and the first detection box that were not matched in the first matching result are respectively matched with Q2 first motion trajectories by intersection-union ratio (IOU) to obtain the second matching result. The Q2 first motion trajectories are the first motion trajectories other than the Q1 first motion trajectory among the M first motion trajectories. The P second motion trajectories are obtained based on the first motion trajectory matched in the first matching result, the first motion trajectory matched in the second matching result, and at least one of the first detection boxes that did not match in the second matching result.

2. The method according to claim 1, characterized in that, The step of detecting target objects in the first image to obtain N first detection boxes and the first feature information of the first image includes: The first image is input into the JDE model to obtain the N first detection boxes and the first feature information output by the JDE model.

3. The method according to claim 2, characterized in that, The JDE model uses convolutional layers to replace the Focus module.

4. The method according to claim 2, characterized in that, At least one activation function in the JDE model is the modified linear unit ReLU activation function.

5. The method according to claim 2, characterized in that, The JDE model is trained using a dataset of training image sequences containing human figures, and the training images are marked with human figure identifiers and bounding boxes.

6. The method according to claim 1, characterized in that, The step of matching the M first motion trajectories with the N first detection boxes and the first feature information to obtain P second motion trajectories includes: Based on the confidence level of the first detection frame, K1 high-confidence detection frames and K2 low-confidence detection frames are obtained from the N first detection frames, wherein the sum of K1 and K2 is less than or equal to N; For each of the K1 high-confidence detection boxes, the high-confidence detection box is matched with the M first motion trajectories to obtain a third matching result; For each of the K2 low-confidence detection boxes, the low-confidence detection box is matched with the first motion trajectory that was not matched in the third matching result to obtain the fourth matching result; Based on the first motion trajectory matched in the third matching result, the first motion trajectory matched in the fourth matching result, and at least one of the high-confidence detection boxes that were not matched in the third matching result, P second motion trajectories are obtained.

7. The method according to claim 6, characterized in that, The method further includes: Predict the j-th bounding box of the j-th trajectory at the current time; The area of ​​the i-th detection box is matched with the area of ​​the j-th detection box to obtain the first matching value between the i-th detection box and the j-th track; Based on the first matching value between the i-th detection box and the j-th track, the target matching result between the i-th detection box and the j-th track is determined; Wherein, if the i-th detection box is the high-confidence detection box and the j-th track is one of the first tracks in the M first motion trajectories, then the target matching result is the third matching result; if the i-th detection box is the low-confidence detection box and the j-th track is a first motion trajectory that is not matched in the third matching result, then the target matching result is the fourth matching result.

8. The method according to claim 7, characterized in that, The step of determining the target matching result between the i-th detection box and the j-th trajectory based on the first matching value between the i-th detection box and the j-th trajectory includes: From the first feature information, determine the first feature value corresponding to the i-th detection box; Extract the second feature value corresponding to the j-th detection box; The distance between the first feature value and the second feature value is determined as the second matching value between the i-th detection box and the j-th track; Based on the first matching value and the second matching value between the i-th detection box and the j-th track, the target matching result between the i-th detection box and the j-th track is determined.

9. The method according to any one of claims 6-8, characterized in that, The step of obtaining P second motion trajectories based on the first motion trajectory matched in the third matching result, the first motion trajectory matched in the fourth matching result, and at least one of the high-confidence detection boxes that did not match in the third matching result includes: The first motion trajectory matched in the third matching result and the first motion trajectory matched in the fourth matching result are updated to obtain the second motion trajectory; Create a new second motion trajectory for the high-confidence detection boxes that are not matched in the third matching result.

10. The method according to claim 1, characterized in that, Determining the user's current movement trajectory within the target space area from the P second movement trajectories includes: The second motion trajectory corresponding to the user's identifier in the P second motion trajectories is determined as the user's motion trajectory within the target space area at the current moment.

11. The method according to any one of claims 1-8 or 10, characterized in that, The method further includes: Based on the user's movement trajectory, determine the time the user stays at the shelf; Based on the time the user spends at the shelf, determine the user's consumption level for the goods on the shelf; Based on the user's spending level on the goods on the shelf and the user's movement trajectory, suspicious bills are traced back to their source.

12. The method according to claim 11, characterized in that, The longer a user stays at the shelf, the higher the user's spending level on the goods on the shelf.

13. A billing processing device, characterized in that, include: The trajectory determination unit is used to determine the user's movement trajectory within the target spatial region at the current moment; The acquisition unit is configured to, based on the user's movement trajectory, acquire the goods consumed by the user in the target space area and acquire the consumption activity information corresponding to the target space area when the user is detected to have entered the checkout area of ​​the target space area; The bill determination unit is used to determine the user's consumption bill based on the goods consumed by the user and the consumption activity information; Specifically, the trajectory determination unit is used to acquire the location information reported by the positioning chip carried by the user, and based on the location information reported by the positioning chip, generate the user's movement trajectory within the target space area at the current moment; or, The system acquires a first image of the user in the target space region captured by the camera at the current moment, and M existing first motion trajectories in the target space region at the current moment, where M is a positive integer; it performs target object detection on the first image to obtain N first detection boxes and first feature information of the first image, and matches the M first motion trajectories with the N first detection boxes based on the N first detection boxes and the first feature information to obtain P second motion trajectories, where the target object includes the user, and N and P are both positive integers; it then determines the user's motion trajectory in the target space region at the current moment from the P second motion trajectories. Specifically, the trajectory determination unit is used to match the M first motion trajectories with the N first detection boxes and the first feature information to obtain P second motion trajectories, including: Predict the second detection box of each of the Q1 first motion trajectories at the current time, and extract the second feature information corresponding to each second detection box. Determine the second feature information corresponding to each of the N first detection boxes from the first feature information. Match the second feature information corresponding to each second detection box with the second feature information corresponding to each first detection box to obtain a first matching result. The Q1 first motion trajectories are the first motion trajectories matched in multiple consecutive frames among the M first motion trajectories. The second feature information corresponding to the first detection box is the feature value corresponding to the center position of the first detection box in the first feature information. The second feature information is the second feature information corresponding to the first detection box matched by the first motion trajectory in the previous frame; the first motion trajectory and the first detection box that were not matched in the first matching result are respectively matched with Q2 first motion trajectories by intersection-union ratio (IOU) to obtain the second matching result, wherein the Q2 first motion trajectories are the first motion trajectories other than the Q1 first motion trajectories among the M first motion trajectories; based on the first motion trajectory matched in the first matching result, the first motion trajectory matched in the second matching result, and at least one of the first detection boxes that were not matched in the second matching result, the P second motion trajectories are obtained.

14. An electronic device, characterized in that, Including processor and memory; The memory is used to store computer programs; The processor is configured to execute the computer program to implement the method as described in any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, Used to store computer programs that cause a computer to perform the method as described in any one of claims 1 to 12.

Images