Abnormal alarm methods, devices, electronic equipment and computer-readable media

Abstract

This disclosure provides embodiments of anomaly alarm methods, apparatuses, devices, and computer-readable media. One specific implementation of the method includes: acquiring a first user image captured at a target exit channel; identifying user identity information corresponding to the first user image; determining whether target value transfer data exists in a value transfer database; in response to the existence of the target value transfer data, generating anomaly verification information corresponding to the user identity information based on the target value transfer data; in response to the anomaly verification result indicating abnormal behavior, generating an anomaly attribution report based on the target value transfer data and the first user image; and performing alarm processing based on the anomaly attribution report targeting the user identity information. This implementation achieves accurate and efficient detection of abnormal behavior by identifying the user identity at the exit channel and linking the value transfer database based on the user identity. Therefore, timely alarm processing is achieved based on the obtained anomaly attribution report.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of computer technology, and more specifically to anomaly alarm methods, apparatus, electronic devices, and computer-readable media. Background Technology

[0002] Currently, effective monitoring and early warning of abnormal behavior of goods are crucial in retail operations. The common approach to detecting and warning of abnormal behavior is through a store-wide ReID tracking system. This system continuously tracks users through a store-wide camera network, establishing a complete trajectory from the shelf to the exit.

[0003] However, when using the above method, the following technical problems often arise: Existing solutions based on store-wide ReID tracking suffer from excessive computational demands. They require processing video stream data from all cameras in the store, placing stringent demands on hardware configurations and resulting in high deployment costs. This makes them difficult to implement in small to medium-sized scenarios, impacting the efficiency of anomaly detection.

[0004] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0006] Some embodiments of this disclosure provide methods, apparatuses, electronic devices, and computer-readable media for anomaly alarms to address one or more of the technical problems mentioned in the background section above.

[0007] In a first aspect, some embodiments of this disclosure provide an anomaly alarm method, including: acquiring a first user image captured at a target exit channel; identifying user identity information corresponding to the first user image; determining whether target value transfer data corresponding to the user identity information exists in a value transfer database, wherein the target value transfer data includes: taking behavior data; in response to the existence, generating anomaly verification information corresponding to the user identity information based on the target value transfer data; in response to the anomaly verification result indicating the existence of anomaly behavior, generating an anomaly attribution report based on the target value transfer data and the first user image; and performing alarm processing for the user identity information based on the anomaly attribution report.

[0008] Secondly, some embodiments of this disclosure provide an anomaly alarm device, including: an acquisition unit configured to acquire a first user image captured at a target exit channel; an identification unit configured to identify user identity information corresponding to the first user image; a determination unit configured to determine whether target value transfer data corresponding to the user identity information exists in a value transfer database, wherein the target value transfer data includes: taking behavior data; a first generation unit configured to generate anomaly verification information corresponding to the user identity information based on the target value transfer data in response to the existence of the target value transfer data; a second generation unit configured to generate anomaly attribution report based on the target value transfer data and the first user image in response to the anomaly verification result indicating the existence of anomaly behavior; and an alarm unit configured to perform alarm processing for the user identity information based on the anomaly attribution report.

[0009] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, such that when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation of the first aspect.

[0010] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method as described in any implementation of the first aspect.

[0011] The above embodiments of this disclosure have the following beneficial effects: The anomaly alarm method of some embodiments of this disclosure achieves accurate and efficient detection of abnormal behavior by identifying the user's identity at the exit channel and linking the value flow database based on the user's identity. Based on this, timely alarm processing is achieved based on the obtained anomaly attribution report. Specifically, the reason why the detection of related abnormal behavior is not accurate and efficient is that existing technical solutions based on whole-store ReID tracking have the technical defect of excessively high computing power requirements. It needs to process video stream data from all cameras in the store, which has stringent hardware requirements, high deployment costs, and is difficult to popularize in small and medium-sized scenarios, thus affecting the efficiency of abnormal behavior detection. Based on this, the anomaly alarm method of some embodiments of this disclosure first acquires a first user image captured at the target exit channel to facilitate subsequent user identity verification. Then, it identifies the user identity information corresponding to the first user image to obtain the corresponding value flow data, and subsequently determines whether the user corresponding to the user identity information has engaged in abnormal behavior based on the value flow data. Secondly, it is determined whether target value transfer data corresponding to the aforementioned user identity information exists in the value transfer database. This target value transfer data includes data on taking actions. Here, the determined user identity information can be used to link the value transfer database, allowing for the determination of abnormal behavior with minimal computational cost based on the value transfer data corresponding to the user identity information. Furthermore, in response to the anomaly verification result indicating abnormal behavior, an anomaly attribution report can be accurately generated from multiple data sources based on the aforementioned target value transfer data and the aforementioned first user image. Finally, based on the anomaly attribution report, alarm processing is executed for the aforementioned user identity information, enabling timely alarm processing with sufficient attribution evidence to prevent item loss. In summary, by identifying user identities at the exit channel and linking the value transfer database based on user identities, accurate and efficient detection of abnormal behavior can be achieved. Thus, timely alarm processing is achieved based on the obtained anomaly attribution report. Attached Figure Description

[0012] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0013] Figure 1 This is a flowchart of some embodiments of the abnormal alarm method according to this disclosure; Figure 2 This is a schematic diagram of the structure of some embodiments of the abnormal alarm device according to the present disclosure; Figure 3This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0014] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0015] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0016] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0017] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0018] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0019] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] refer to Figure 1 The diagram illustrates a flow 100 of some embodiments of an anomaly alarm method according to the present disclosure. This anomaly alarm method includes the following steps: Step 101: Obtain the first user image captured at the target exit channel.

[0021] In some embodiments, the executing entity of the above-described anomaly alarm method (e.g., an electronic device) can acquire a first user image captured at the target exit channel via wired or wireless means. The target exit channel can be the exit channel of the target store where anomaly attribution is performed. Here, anomaly attribution can be the attribution of damage to items being sold in the target store (i.e., determining the cause of abnormal damage to the items). For example, abnormal damage can be one of the following: loss or damage. That is, by capturing real-time user images at the target exit channel, the loss or abnormal damage of items in the target store can be prevented. In practice, the target store can be a store where items are sold. The items for anomaly attribution can be high-value goods sold in the target store. For example, the item for anomaly attribution can be gold. The target exit channel can be a specific exit channel in the target store. The first user image can be an image displaying user information. The first user image displays an image of a departing customer.

[0022] As an example, the aforementioned entity could invoke the deployment of a high-speed global camera at the target exit channel to acquire the first user image. Here, by deploying a global camera, it is ensured that a clear user image can be captured even when the user is moving rapidly.

[0023] Step 102: Identify the user identity information corresponding to the first user image mentioned above.

[0024] In some embodiments, the executing entity may identify user identity information corresponding to the first user image. The user identity information may be the identity information of the user displayed in the first user image. For example, the user identity information may be a user identifier. For example, the user identity information may be "01023#". The user identity information may also be an anonymous identifier.

[0025] As an example, the aforementioned executing entity can invoke the facial recognition model deployed corresponding to the target exit channel to identify the user's identity in the first user image, thereby determining the anonymous identity identifier corresponding to the user's identity information as the user's identity information.

[0026] Step 103: Determine whether target value transfer data corresponding to the aforementioned user identity information exists in the value transfer database.

[0027] In some embodiments, the aforementioned executing entity may determine whether target value transfer data corresponding to the aforementioned user identity information exists in the value transfer database. The value transfer database may be a database storing value transfer data. Here, value transfer data may be data on user purchases and sales of goods. In practice, value transfer data may include: user identity information, purchase and sale price, user image, retrieval behavior data, value transfer time, and image capture time. The purchase and sale price may be the price at which the user buys and sells the target item in the target store. The user image may be a first user image. The retrieval behavior data may be the behavior data of the user corresponding to the user identity information retrieving the target item from the shelf area. In practice, retrieval behavior data may include: shelf area, retrieval time, retrieval action sequence, and user identity information. The shelf area may be the area where the user corresponding to the user identity information retrieves the target item. The retrieval time may be the time when the user corresponding to the user identity information retrieves the target item to view it or perform other actions. The retrieval action sequence may be a series of actions performed by the user corresponding to the user identity information in retrieving the target item. For example, the retrieval action sequence may include: reaching out to touch, reaching out to view, grasping and holding, displacement and movement, returning the item, and retracting the hand. The user identity information in the target value transfer data is the user identity information corresponding to the first user image.

[0028] As an example, the aforementioned executing entity can use query statements to determine whether target value transfer data corresponding to the aforementioned user identity information exists in the value transfer database.

[0029] In some optional implementations of certain embodiments, the value transfer data in the aforementioned value transfer database is generated through the following steps: The first step, in response to the detection of a value transfer by a first user, is to extract user feature information from the image of the second user corresponding to that first user. In practice, the value transfer can be a payment operation. The first user is the user performing the payment operation. The second user image can be a user image captured from the first user. In specific practice, when the payment system detects that a first user is performing a payment operation, the second user image can be obtained through a camera device deployed at the payment terminal (i.e., the cashier). The user feature information can be a vector representing high-quality biometric semantics. In practice, the user feature information can be a 512-dimensional feature vector.

[0030] As an example, the aforementioned execution entity can utilize a multimodal large model to extract high-quality biometric features from the second user image, thereby obtaining user feature information. The multimodal large model can be a large model that supports high-quality biometric feature extraction.

[0031] The second step is to filter out target taking behavior data that has not yet undergone value transfer from the taking behavior database, thus obtaining the target taking behavior dataset. The taking behavior database can store taking behavior data of various users within the target store. "Not yet undergone value transfer" means that no payment has been made. The target taking behavior data here refers to the behavior data of users who have taken items but have not yet paid for them.

[0032] The third step is to determine the semantic content similarity between each target retrieval behavior data point in the aforementioned target retrieval behavior dataset and the aforementioned user feature information for each target retrieval behavior data point. Specifically, the semantic content similarity can be the semantic content similarity between the semantic content of the user taking the action in the target retrieval behavior data and the semantic content of the user feature information. The semantic content similarity can be a value between 0 and 1; a higher value indicates a higher similarity in the feature content of the two.

[0033] As an example, firstly, user data for the target user is obtained from the target user behavior data. Then, the user data is vectorized to obtain the user feature information. Next, the cosine similarity between the user feature information and the target user feature information is determined as the feature semantic content similarity.

[0034] The fourth step involves determining the existence of target acquisition behavior data with a semantic content similarity higher than the target similarity. Based on this data and the value transfer time, value transfer data is generated for the aforementioned first user. The target similarity can be a pre-set threshold. Target similarity is a numerical value that measures whether semantic content is highly similar. For example, the target similarity could be 0.8. The value transfer time can be the payment time.

[0035] As an example, the aforementioned executing entity can combine target acquisition behavior data with higher similarity to the target, value transfer time, and user identity information corresponding to the first user to obtain value transfer data.

[0036] In some optional implementations of certain embodiments, the take-up behavior data in the above-mentioned take-up behavior database is generated through the following steps: The first step is to acquire real-time behavioral video of the second user, captured in the shelf area. The shelf area can be the area where the items displayed in the target store are located. The second user can be any user within the shelf area who is likely to buy or sell items. The real-time behavioral video can be video footage of the second user's actions within the shelf area, captured in real-time.

[0037] Here, the items displayed on the shelves can be high-value items. That is, the shelf area can be a high-value merchandise area. A dedicated array of recognition cameras is deployed in the shelf area, using an optimal installation angle of 30-45 degrees to ensure clear capture of hand-to-product interaction details. The cameras are selected with at least 2 megapixels, supporting 1080P resolution and a capture rate of 25fps, and equipped with wide dynamic range processing capabilities to adapt to different lighting conditions.

[0038] The second step is to determine the object boundary information and user keypoint information corresponding to each frame of the real-time behavior video. The behavior image can be a single frame from the real-time behavior video. The object boundary information can be the boundary range of the object grasped or targeted by the second user in the behavior image. In practice, the object boundary information can be in coordinate form. The user keypoint information can be the human skeletal keypoints corresponding to the second user in the behavior image. In practice, the user keypoint information can also be displayed in pixel coordinates.

[0039] As an example, a multimodal large model can be used to determine the object boundary information and user key point information corresponding to each frame of behavioral image.

[0040] The third step involves generating the spatial relationship between the user's hand key points and the item boundaries based on the obtained item boundary information sequence and user key point information sequence. The item boundary information sequence represents the location information of the user picking up the item in the real-time behavioral video. The user key point information sequence represents the second user's behavioral actions within the shelf area. The spatial relationship can be one of the following: a spatial relationship with contact, or a spatial relationship without contact.

[0041] As an example, for each item boundary information in the item boundary information sequence, firstly, the aforementioned execution entity can determine the user keypoint information with the same existence time corresponding to the item boundary information, as the target user keypoint information. Then, it determines whether the coordinate range corresponding to the aforementioned item boundary information overlaps with the range of keypoints related to the hand in the user keypoint information, thus obtaining overlap information. In response to determining that multiple overlapping information representations have overlapping ranges, a spatial relationship where the representations are in contact is generated. In response to determining that the overlapping information representations do not have overlapping ranges, a spatial relationship where the representations are not in contact is generated.

[0042] Fourth, in response to the aforementioned spatial relationship satisfying the target relationship condition, based on the aforementioned item boundary information sequence and the aforementioned user key point information sequence, generate behavioral completeness information representing whether each action corresponding to the retrieval behavior is complete. The target relationship condition can be a spatial relationship representing the existence of contact. Each action can include: reaching out to contact, grasping and holding, displacement and movement, and hand retraction. For the retrieval action, the corresponding actions have a sequence, which is {reaching out to contact -> grasping and holding -> displacement and movement -> hand retraction}. Behavioral completeness information can be whether each action included in the retrieval behavior is a reaching out to contact, grasping and holding, displacement and movement, or hand retraction action. Behavioral completeness information can be one of the following: information representing the completeness of each action corresponding to the retrieval behavior, or information representing the incompleteness of each action corresponding to the retrieval behavior.

[0043] As an example, the aforementioned executing entity can use a multimodal large model to determine whether the actions reflected in the user key point information sequence are complete behavioral information of each action in the complete picking behavior, based on the item boundary information sequence and the user key point information sequence.

[0044] Fifth, in response to the complete information representation of the above-mentioned behavior, the user identity information and user feature vector corresponding to the second user are extracted. The user identity information represents the identity of the second user. In practice, the user identity information can be an anonymous user identifier corresponding to the second user. The user feature vector represents a vector of semantic content of the user features corresponding to the second user.

[0045] As an example, firstly, the frame images that best represent the characteristics of the second user can be selected from the real-time behavioral video. Then, using a deep convolutional neural network, user feature vectors are extracted from the frame images. Using an encryption algorithm, the user feature vectors are encrypted to obtain a globally unique anonymous identity identifier, which serves as the user's identity information.

[0046] Step 6: Based on the user identity information corresponding to the second user, the corresponding item information, the user feature vector, and the corresponding video shooting location, generate the retrieval behavior data. The item information can be the item information of the item retrieved by the second user in the real-time behavior video. For example, the item information can be the item representation of the retrieved item. The video shooting location can be the actual location of the camera that captured the real-time behavior video.

[0047] As an example, the aforementioned executing entity can combine the user identity information corresponding to the second user, the corresponding item information, the aforementioned user feature vector, and the corresponding video shooting location to obtain the taking behavior data.

[0048] In some optional implementations of certain embodiments, the aforementioned retrieval behavior data is generated based on edge computing devices deployed in dedicated shelf areas; value flow data is generated based on store servers deployed in value flow areas whose performance meets target conditions; and anomaly attribution reports are generated based on store servers deployed in the aforementioned target exit channels whose performance meets target conditions. Edge computing devices can be computing devices deployed in edge areas (e.g., shelf areas) to process related computing tasks in the shelf area. Target conditions can be performance indicators used to measure whether a server is a high-performance server. Store servers can be servers deployed within the store. Target conditions can include: storage performance conditions, CPU performance conditions, GPU / accelerator performance conditions, and network performance conditions. Storage performance conditions can be conditions requiring high storage performance. CPU performance conditions can be conditions requiring high CPU performance. GPU / accelerator performance conditions can be conditions requiring high GPU / accelerator performance. Network performance conditions can be conditions requiring high network performance.

[0049] Step 104: In response to existence, generate abnormal verification information corresponding to the above-mentioned user identity information based on the above-mentioned target value flow data.

[0050] In some embodiments, in response to existence, the aforementioned executing entity can generate anomaly verification information corresponding to the aforementioned user identity information based on the aforementioned target value flow data. The anomaly verification information can be verification information indicating whether the user corresponding to the user identity information exhibits abnormal behavior. Specifically, the anomaly verification information can be one of the following: information indicating that the user corresponding to the user identity information exhibits abnormal behavior, or information indicating that the user corresponding to the user identity information does not exhibit abnormal behavior.

[0051] As an example, firstly, the aforementioned executing entity determines whether the target value flow data meets the various anomaly verification rules in a pre-set anomaly verification rule set, thus obtaining a satisfaction result. Anomaly verification rules can be rules set to verify anomalies in various aspects of the buying and selling of the target item. Each anomaly verification rule can be pre-set. Then, in response to the satisfaction result indicating that the various anomaly verification rules are met, information indicating that the user corresponding to the user's identity information does not exhibit abnormal behavior is generated. In response to the satisfaction result indicating that the various anomaly verification rules are not met, information indicating that the user corresponding to the user's identity information exhibits abnormal behavior is generated.

[0052] In some optional implementations of certain embodiments, the aforementioned execution entity may generate abnormal verification information corresponding to the aforementioned user identity information based on the aforementioned target value flow data, including the following steps: The first step is to verify whether the target item exists in the item list within the aforementioned target value flow data, thus obtaining the first verification result. The item list can be a list of all items paid for or taken by the user within the target value flow data. The target item can be an item taken using the user's identity information captured at the target exit channel. In practice, the target item can be packaging of the target brand or the actual item taken. The first verification result can be one of the following: a verification result indicating that the target item does not exist in the item list, or a verification result indicating that the target item exists in the item list. The item list can include: all paid items and all items taken.

[0053] As an example, for a target item that is the packaging of a target brand, it can be verified whether each item in the item list corresponds to the same brand as the target item. For a target item that is actually taken, it can be verified whether the target item exists in each item in the item list and whether each brand is the same as the brand corresponding to the target item.

[0054] The second step is to verify the logical consistency between the taking time in the aforementioned taking behavior data and the value transfer time in the aforementioned target value transfer data, thus obtaining a second verification result. Here, the value transfer time can be the payment time corresponding to each paid item in the item list. Logical consistency means that for paid items, the taking time is earlier than the value transfer time. The second verification result can be one of the following: a verification result indicating logical consistency in time, or a verification result indicating logical inconsistency in time.

[0055] The third step is to generate abnormal verification information based on the first verification result and the second verification result.

[0056] As an example, in response to determining that at least one of the first and second verification results indicates that the verification is abnormal (e.g., the timing logic is unreasonable or the target item does not exist), a verification result indicating abnormal behavior is generated.

[0057] Step 105: In response to the above-mentioned anomaly verification results indicating the existence of abnormal behavior, an anomaly attribution report is generated based on the above-mentioned target value flow data.

[0058] In some embodiments, in response to the above-mentioned anomaly verification results indicating the existence of abnormal behavior, the executing entity can generate an anomaly attribution report based on the above-mentioned target value flow data. The abnormal behavior can be one of the following: anomaly of carrying the item out of the store without payment, or anomaly of malicious damage after purchase. The anomaly attribution report can be a report summarizing the causes of the abnormal behavior. The anomaly attribution report may include: user characteristic image, video clip of the taking behavior, payment verification result, exit capture evidence, timeline analysis, and confidence score. The user characteristic image can be a first user image. The video clip of the taking behavior can be a video clip of a user taking the target item corresponding to the user's identity information in the taking behavior data. The payment verification result can be the verification result of the user's payment operation process in the value flow data. The exit capture evidence can be evidence of abnormal behavior captured at the target exit channel. The timeline analysis can be the timeline analysis result of various events corresponding to the user's identity information purchasing the target item, making payment, and leaving the target store. The confidence score can be the accuracy of the anomaly attribution report.

[0059] As an example, the aforementioned implementing entity can utilize a multimodal big data model to generate anomaly attribution reports based on target value flow data. Here, anomaly attribution prompts can be generated to guide the multimodal big data model in generating the anomaly attribution report.

[0060] In some optional implementations of certain embodiments, after step 105, the steps further include: In response to the above-mentioned anomaly verification result indicating the absence of abnormal behavior, the aforementioned executing entity can remove the status detection tag corresponding to the user identity information, as well as the temporary storage data corresponding to the user identity information. The status detection tag can be a tag indicating whether abnormal behavior monitoring is performed on the user corresponding to the user identity information. In practice, when a user has a status detection tag, it indicates that abnormal behavior monitoring is performed on the user. When a user does not have a status detection tag, it indicates that abnormal behavior monitoring is not performed on the user. The temporary storage data can be a captured image or payment data corresponding to the user identity information. Here, the temporary storage data can be data related to the user corresponding to the user identity information that is temporarily stored.

[0061] In some optional implementations of certain embodiments, generating an anomaly attribution report based on the target value flow data and the first user image includes the following steps: The first step, in response to the existence of the target item in the aforementioned target value flow data, involves obtaining at least one route information corresponding to the user's identity information from the store aisle map based on the shelf area location corresponding to the target item and the target exit aisle in the target value flow data. The shelf area location can be the location of the shelf area where the paid item is stored. The store aisle map can be a schematic diagram representing each driving aisle in the store layout corresponding to the target store. The shelf area location corresponding to the target item can be the location information of the shelf where the item is placed. The user's identity information-related movement can be the walking movement of the user corresponding to the user's identity information. The route information in at least one route can be a preliminary predicted route for the walking movement of the user corresponding to the user's identity information. The starting point of each route information is the shelf area location, and the corresponding ending point is the target exit aisle.

[0062] As an example, the location of the shelf area can be used as the starting point and the target exit aisle as the ending point. Route information including the location of the shelf area and the target exit aisle can be selected from the various route information marked on the store aisle map and used as at least one route information corresponding to the user's identity information.

[0063] The second step is to sort at least one of the above route information based on the pedestrian flow information corresponding to each route information, thereby obtaining a route information sequence. The pedestrian flow information can be the average pedestrian flow per minute. The sequence order of the various route information items in the route information sequence is based on the pedestrian flow rate from highest to lowest.

[0064] The third step is to perform the following generation steps for the target route information in the route information sequence: Sub-step 1 involves determining whether the abnormal behavior database for detecting abnormal behavior on the target route information contains route abnormal behavior data related to the user corresponding to the aforementioned user identity information. The target route information can be the route information currently being determined to be the movement of the user corresponding to the user identity information. The route abnormal behavior database can be a database storing abnormal behavior data for each route. The route abnormal behavior data can be behavioral data of abnormal behaviors existing within the route. For example, route abnormal behavior data can include: abnormal behavior type, user identity information, abnormal behavior time, and abnormal behavior location. In practice, the route abnormal behavior data in the route abnormal behavior database can be generated in real-time based on behavioral events occurring on the target route information.

[0065] Sub-step 2, in response to the determination that it does not exist, determines the identity binding database corresponding to the key road segment of the aforementioned target route information. The key road segment can be the road segment or intersection with the highest pedestrian traffic in the target route information. The identity binding database can be a database storing various identity binding data. The identity binding data can be the identity data of users who have traveled through the key road segment. For example, the identity binding data can include: user identity identifiers and user images.

[0066] Sub-step 3: Determine whether there is identity binding information corresponding to the aforementioned user identity information in the identity binding database. Specifically, identity binding information corresponding to the user identity information is where the user identity identifier corresponding to the user identity information is the same as the user identity identifier in the identity binding information.

[0067] As an example, a query can be used to determine whether the aforementioned identity binding database contains identity binding information corresponding to the aforementioned user identity information.

[0068] Sub-step 4, in response to the determination of existence, determines at least one identity binding database corresponding to at least one secondary key road segment for the aforementioned target route information. The secondary key road segment can be a predetermined number of road segments with the highest pedestrian traffic, second only to the key road segment. For example, the predetermined number could be 4. Each secondary key road segment has a unique corresponding identity binding database.

[0069] Sub-step 5 involves determining whether at least one of the aforementioned identity binding databases contains identity binding information corresponding to the aforementioned user identity information. Further details are omitted.

[0070] Sub-step 6: In response to the determination that all exist, generate a driving route that represents the target route information as the user corresponding to the user identity information.

[0071] Sub-step 7 involves filtering motion videos related to the users corresponding to the aforementioned user identity information from the video sequences corresponding to the route information sequence. There is a one-to-one correspondence between the route information in the route information sequence and the video sequences in the video sequences.

[0072] Sub-step 8: Based on the above motion video, generate abnormal behavior data of the target route targeting the above user identity information.

[0073] As an example, the aforementioned executing entity can utilize the abnormal behavior detection function of the multimodal large model to generate abnormal behavior data of the target route based on the aforementioned user identity information.

[0074] Specifically, for the abnormal behavior detection function, a behavior dataset can be used to train an initial multimodal large model, resulting in a multimodal large model. Behavioral data can include behavioral videos and abnormal labels. Behavioral videos can serve as training data, while abnormal labels are the training targets.

[0075] The fourth step is to generate an anomaly attribution report based on the aforementioned abnormal behavior data of the target route, the aforementioned target value transfer data, and the aforementioned first user image.

[0076] As an example, the aforementioned executing entity can combine the aforementioned abnormal behavior data of the target route, the aforementioned target value transfer data, and the aforementioned first user image in a predetermined format to obtain an anomaly attribution report.

[0077] Fifth, in response to the absence of identity binding information corresponding to the aforementioned user identity information in at least one of the aforementioned identity binding databases, determine the next route information in the route information sequence corresponding to the aforementioned target route information. The next route information may be the next route information whose sequence position is adjacent to the target route information.

[0078] Step 6: Use the next route information as the target route information and continue with the above generation steps.

[0079] Step 7: Based on the generation time corresponding to the target value flow data and the shooting time corresponding to the first user image, remove video segments whose shooting time is earlier than the generation time and later than the shooting time from the video sequence to obtain the cropped video sequence.

[0080] Step 8: Deduplication and fusion processing are performed on the same video segments in the above cropped video sequence to obtain the fused video.

[0081] As an example, the aforementioned execution entity can use an automatic video cropping tool to deduplicate and merge identical video segments in the cropped video sequence to obtain a merged video.

[0082] Step 9: Based on at least one generation time of the corresponding identity binding information generated from the above-mentioned identity binding database and the generation time of the corresponding identity binding information generated from the above-mentioned identity binding database, perform key segment cropping on the above-mentioned fused video to obtain a key segment sequence, which is used as motion video.

[0083] Optionally, the above-mentioned generation of abnormal behavior data for the target route based on the above-mentioned motion video and the above-mentioned user identity information includes the following steps: The first step is to generate local abnormal behavior data for each key segment by using the store servers deployed with performance that meet the target conditions corresponding to the above target route information.

[0084] The second step is to identify the obtained local abnormal behavior data of each route as the abnormal behavior data of the target route.

[0085] Optionally, the above method further includes: The first step, in response to the absence of the target item in the aforementioned target value flow data, involves reverse tracking of the video frames corresponding to the user identity information based on the first user image, thereby obtaining the motion video corresponding to the user identity information. Specifically, reverse tracking can be performed by using the user's pose corresponding to the first user image to perform forward human body tracking on the video corresponding to the first user image, thus obtaining the motion video related to the user identity information.

[0086] The second step involves generating abnormal movement behavior data corresponding to the user's identity information based on the aforementioned motion videos. Details will not be elaborated further.

[0087] The third step involves generating an anomaly attribution report based on the aforementioned abnormal movement behavior data, target value transfer data, and the first user image. Details will not be elaborated further.

[0088] The above embodiments of this disclosure have the following beneficial effects: The anomaly alarm method of some embodiments of this disclosure achieves accurate and efficient detection of abnormal behavior by identifying the user's identity at the exit channel and linking the value flow database based on the user's identity. Based on this, timely alarm processing is achieved based on the obtained anomaly attribution report. Specifically, the reason why the detection of related abnormal behavior is not accurate and efficient is that existing technical solutions based on whole-store ReID tracking have the technical defect of excessively high computing power requirements. It needs to process video stream data from all cameras in the store, which has stringent hardware requirements, high deployment costs, and is difficult to popularize in small and medium-sized scenarios, thus affecting the efficiency of abnormal behavior detection. Based on this, the anomaly alarm method of some embodiments of this disclosure first acquires a first user image captured at the target exit channel to facilitate subsequent user identity verification. Then, it identifies the user identity information corresponding to the first user image to obtain the corresponding value flow data, and subsequently determines whether the user corresponding to the user identity information has engaged in abnormal behavior based on the value flow data. Secondly, it is determined whether target value transfer data corresponding to the aforementioned user identity information exists in the value transfer database. This target value transfer data includes data on taking actions. Here, the determined user identity information can be used to link the value transfer database, allowing for the determination of abnormal behavior with minimal computational cost based on the value transfer data corresponding to the user identity information. Furthermore, in response to the anomaly verification result indicating abnormal behavior, an anomaly attribution report can be accurately generated from multiple data sources based on the aforementioned target value transfer data and the aforementioned first user image. Finally, based on the anomaly attribution report, alarm processing is executed for the aforementioned user identity information, enabling timely alarm processing with sufficient attribution evidence to prevent item loss. In summary, by identifying user identities at the exit channel and linking the value transfer database based on user identities, accurate and efficient detection of abnormal behavior can be achieved. Thus, timely alarm processing is achieved based on the obtained anomaly attribution report.

[0089] Further reference Figure 2 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of an anomaly alarm device, which are similar to... Figure 1 Corresponding to the method embodiments shown, this anomaly alarm device can be specifically applied to various electronic devices.

[0090] like Figure 2As shown, an anomaly alarm device 200 includes: an acquisition unit 201, an identification unit 202, a determination unit 203, a first generation unit 204, and a second generation unit 205. The acquisition unit 201 is configured to acquire a first user image captured at a target exit channel; the identification unit 202 is configured to identify user identity information corresponding to the first user image; the determination unit 203 is configured to determine whether target value transfer data corresponding to the user identity information exists in a value transfer database, wherein the target value transfer data includes: taking behavior data; the first generation unit 204 is configured to generate anomaly verification information corresponding to the user identity information based on the target value transfer data in response to the existence of such information; the second generation unit 205 is configured to generate anomaly attribution report based on the target value transfer data and the first user image in response to the anomaly verification result indicating the existence of anomaly behavior; and the alarm unit 206 is configured to perform alarm processing for the user identity information based on the anomaly attribution report.

[0091] It is understandable that the units described in the abnormal alarm device 200 are related to the reference. Figure 1 The steps in the described method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the abnormal alarm device 200 and the units contained therein, and will not be repeated here.

[0092] The following is for reference. Figure 3 It shows a schematic diagram of the structure of an electronic device (e.g., an electronic device) 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0093] like Figure 3 As shown, the electronic device 300 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the electronic device 300. The processing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0094] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 An electronic device 300 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 3 Each box shown can represent a device or multiple devices as needed.

[0095] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 309, or installed from storage device 308, or installed from ROM 302. When the computer program is executed by processing device 301, it performs the functions defined in the methods of some embodiments of this disclosure.

[0096] It should be noted that, in some embodiments of this disclosure, the computer-readable medium described above may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0097] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0098] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently without being assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: acquire a first user image captured at the target exit channel; identify user identity information corresponding to the first user image; determine whether target value transfer data corresponding to the user identity information exists in the value transfer database, wherein the target value transfer data includes: taking behavior data; in response to the existence of the target value transfer data, generate abnormal verification information corresponding to the user identity information based on the target value transfer data; and in response to the abnormal verification result indicating the existence of abnormal behavior, generate an abnormal attribution report based on the target value transfer data.

[0099] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0100] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0101] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including an acquisition unit, an identification unit, a determination unit, a first generation unit, and a second generation unit. The names of these units do not necessarily limit the specific unit; for example, the acquisition unit may also be described as "a unit for acquiring a first user image captured at a target exit channel."

[0102] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0103] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. An abnormal alarm method, comprising: Acquire the first user image captured at the target exit channel; Identify the user identity information corresponding to the first user image; Determine whether target value transfer data corresponding to the user's identity information exists in the value transfer database, wherein the target value transfer data includes: taking behavior data; In response to the existence, based on the target value flow data, anomaly verification information corresponding to the user identity information is generated; In response to the anomaly verification result indicating the presence of abnormal behavior, an anomaly attribution report is generated based on the target value flow data and the first user image; Based on the anomaly attribution report, perform alarm processing for the user's identity information.

2. The method according to claim 1, wherein, The step of generating abnormal verification information corresponding to the user identity information based on the target value flow data includes: Verify whether the target item exists in the item list in the target value flow data to obtain the first verification result; Verify the logical consistency between the taking time in the taking behavior data and the value transfer time in the target value transfer data to obtain a second verification result; Based on the first verification result and the second verification result, anomaly verification information is generated.

3. The method according to claim 1, wherein, The method further includes: In response to the anomaly verification result indicating the absence of abnormal behavior, the status detection tag corresponding to the user identity information is removed, as well as the temporary storage data corresponding to the user identity information is removed.

4. The method according to claim 1, wherein, The value transfer data in the value transfer database is generated through the following steps: In response to detecting a value transfer by a first user, user feature information is extracted from the image of the second user corresponding to the first user. The target taking behavior dataset is obtained by filtering out target taking behavior data that has not undergone value transfer from the taking behavior database; For each target retrieval behavior data in the target retrieval behavior dataset, determine the feature semantic content similarity between the target retrieval behavior data and the user feature information; In response to the determination that there is target retrieval behavior data with semantic content similarity higher than target similarity, value transfer data for the first user is generated based on the target retrieval behavior data with higher similarity than target and the value transfer time.

5. The method according to claim 4, wherein, The retrieval behavior data in the retrieval behavior database is generated through the following steps: Acquire real-time behavioral video of a second user captured in the shelf area; Determine the object boundary information and user key point information corresponding to each frame of the real-time behavior video; Based on the obtained sequence of object boundary information and sequence of user key points information, the spatial relationship between the user's hand key points and the object boundary is generated; In response to the spatial relationship satisfying the target relationship condition, behavioral completeness information representing whether each action corresponding to the retrieval behavior is complete is generated based on the item boundary information sequence and the user key point information sequence; In response to the completeness of the behavior information representation, corresponding to each action of the retrieval behavior is complete, the user identity information and user feature vector corresponding to the second user are extracted; Based on the user identity information corresponding to the second user, the corresponding item information, the user feature vector, and the corresponding video shooting location, take-up behavior data is generated.

6. The method according to claim 5, wherein, The retrieval behavior data is generated based on edge computing devices deployed in the dedicated shelf area, the value flow data is generated based on store servers deployed in the value flow area whose performance meets the target conditions, and the anomaly attribution report is generated based on store servers deployed in the target exit channel whose performance meets the target conditions.

7. The method according to claim 1, wherein, The step of generating an anomaly attribution report based on the target value flow data and the first user image includes: In response to the presence of the target item in the target value flow data, based on the shelf area location corresponding to the target item in the target value flow data and the target exit channel, at least one route information corresponding to the user's identity information is obtained from the store channel map; Based on the pedestrian flow information corresponding to each route information, the at least one route information is sorted to obtain a route information sequence; For the target route information in the route information sequence, perform the following generation steps: Determine whether there is any abnormal route behavior data related to the user corresponding to the user identity information in the abnormal route behavior database corresponding to the target route information; In response to the determination that the target route information does not exist, the identity binding database corresponding to the key road segment is determined. Determine whether identity binding information corresponding to the user's identity information exists in the identity binding database; In response to the determination of existence, it is determined that the target route information corresponds to at least one identity binding database corresponding to at least one secondary key road segment; Determine whether identity binding information corresponding to the user's identity information exists in the at least one identity binding database; In response to the determination that both exist, a driving route is generated that represents the target route information as the driving route of the user corresponding to the user identity information; Select motion videos related to the user corresponding to the user identity information from the video sequence corresponding to the route information sequence; Based on the motion video, generate abnormal behavior data of the target route targeting the user's identity information; An anomaly attribution report is generated based on the target route abnormal behavior data, the target value flow data, and the first user image. In response to the absence of identity binding information corresponding to the user identity information in at least one identity binding database, the next route information in the route information sequence corresponding to the target route information is determined; Use the next route information as the target route information and continue with the generation step; In response to the absence of the target item in the target value flow data, the video frame corresponding to the user identity information is traced in reverse based on the first user image to obtain the motion video corresponding to the user identity information; Based on the motion video, generate abnormal motion behavior data corresponding to the user's identity information; An anomaly attribution report is generated based on the abnormal movement behavior data, the target value transfer data, and the first user image.

8. An abnormal alarm device, comprising: The acquisition unit is configured to acquire a first user image captured at the target exit channel; The identification unit is configured to identify the user identity information corresponding to the first user image; The determining unit is configured to determine whether target value transfer data corresponding to the user identity information exists in the value transfer database, wherein the target value transfer data includes: taking behavior data; The first generation unit is configured to generate abnormal verification information corresponding to the user identity information in response to the existence of the target value flow data. The second generation unit is configured to generate an anomaly attribution report in response to the anomaly verification result indicating the existence of anomaly behavior, based on the target value flow data and the first user image. The alarm unit is configured to perform alarm processing for the user identity information based on the anomaly attribution report.

9. An electronic device, comprising: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-7.

10. A computer-readable medium having a computer program stored thereon, wherein, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.

Images