Abnormal operation identification method, model construction method and computer program product

Abstract

The embodiment of the invention discloses an abnormal operation identification method, a model construction method and a computer program product. The method comprises the following steps: obtaining an operation video of a commodity code scanning settlement process performed by a consumer user through self-service cash register equipment; target area identification is carried out on image frames in the operation video; a commodity recognition model is obtained, the image frame of the target area is recognized to serve as input, and a first recognition result output by the commodity recognition model is obtained and comprises position information of the to-be-settled commodity in the image frame when a user holds the to-be-settled commodity to execute code scanning operation; extracting commodity feature information of the to-be-settled commodity from the image frame according to the position information, and performing commodity deduplication processing according to the commodity feature information to obtain the number of code-scanned commodities; and determining the number of settled commodities in the transaction order corresponding to the operation video, and performing abnormal operation judgment on the user according to the matching condition of the number of the scanned commodities and the number of the settled commodities. The abnormal operation in the commodity code scanning process can be simply and conveniently identified.

Description

Technical Field

[0001] The present application relates to the technical field of commodity information services, and in particular to an abnormal operation recognition method and device, a commodity recognition model construction method and device, a behavior recognition model construction method and device, a computer-readable storage medium, an electronic device, and a computer program product. Background Art

[0002] To improve the shopping settlement efficiency of offline stores, cash register devices such as POS machines are set up in the stores for consumers to scan codes and settle accounts by themselves. In actual applications, there are often intentional or unintentional abnormal operations such as missed scans, which cause losses to the stores.

[0003] To solve this problem, most stores will arrange full-time personnel in the self-checkout area to conduct real-time inspections on the code-scanning behaviors of users, and promptly discover and correct abnormal operations. This not only increases the store costs, but also may affect the user experience of self-checkout.

[0004] In addition, there are also some automatic recognition solutions for abnormal operations. For example, a weighing component is installed in the cash register device to confirm the commodities scanned by the user in a weighing manner. For another example, the operation table of the cash register device is transformed and divided into an area for unscanned commodities and an area for scanned commodities. The user's code-scanning video can be analyzed. If the user's hand shows a movement trajectory from the area for unscanned commodities to the area for scanned commodities, but the corresponding commodity barcode is not recognized by the cash register device, that is, the scanned commodity is not entered into the system, then this code-scanning behavior can be determined as an abnormal operation.

[0005] The above-mentioned automatic recognition solutions all involve hardware transformation and upgrading of the cash register device, and also need to restrict the code-scanning methods used by users when scanning commodities. That is to say, different recognition solutions correspond to different code-scanning methods, and users need to use the corresponding code-scanning methods to conduct self-checkout and settlement according to the recognition solutions supported by the POS machine. For example, in the weighing method, it is required that after the user scans the commodity barcode, the commodity should be placed on the bagging table equipped with the weighing component, so that after the commodity weighing is completed, the POS machine system can verify the commodity weight according to the preset weighing anti-loss verification rules, and add the scanned commodity to the shopping list when the commodity weight meets the verification rules. For another example, in the cross-zone method, it is required that during the process of the user scanning the commodity barcode, the user must hold the commodity and make a movement trajectory from the area for unscanned commodities to the area for scanned commodities.

[0006] How to simply and conveniently recognize abnormal operations during the commodity code-scanning process has become a technical problem that needs to be solved by those skilled in the art. Summary of the Invention

[0007] The present application provides an abnormal operation recognition method and apparatus, a commodity recognition model construction method and apparatus, a behavior recognition model construction method and apparatus, a computer-readable storage medium, an electronic device, and a computer program product, which can simply and conveniently recognize abnormal operations during the commodity scanning process.

[0008] The present application provides the following solutions:

[0009] An abnormal operation recognition method, the method comprising:

[0010] Obtaining an operation video of a consumer user during the process of scanning and settling commodities through a self-checkout device;

[0011] Performing target area recognition on the image frames in the operation video, where the target area includes an image area related to the operation of the consumer user for scanning and settling commodities;

[0012] Obtaining a commodity recognition model, and using the image frames in which the target area is recognized as input to obtain a first recognition result output by the commodity recognition model, where the first recognition result includes the position information of the to-be-settled commodity in the image frame when the consumer user holds the to-be-settled commodity and performs a scanning operation;

[0013] Extracting the commodity feature information of the to-be-settled commodity from the image frame according to the position information, and then performing duplicate commodity processing according to the commodity feature information to obtain the number of scanned commodities of the consumer user;

[0014] Determining the number of settled commodities in the transaction order according to the transaction order corresponding to the operation video, and judging whether the consumer user has abnormal operations according to the matching situation between the number of scanned commodities and the number of settled commodities.

[0015] Wherein, the method further comprises:

[0016] Obtaining a gesture recognition model, and using the image frames in which the target area is recognized as input to obtain a second recognition result output by the gesture recognition model, where the second recognition result is used to indicate whether a gesture of holding a commodity is detected in the image frame, and the gesture is recognized by the gesture recognition model by marking the key points of the human hand joints after the human hand of the consumer user is recognized in the target area;

[0017] Using the image frames in which a gesture of holding a commodity is detected as the input of the commodity recognition model to obtain the first recognition result corresponding to the output of the commodity recognition model.

[0018] Wherein, the second recognition result is further used to indicate whether a swiping gesture for operating the self-service cash register is detected in the image frame. The swiping gesture is recognized by the gesture recognition model by annotating the key points of the human hand joints. The method further includes:

[0019] Determine the operation time when the consumer user executes the swiping gesture, and obtain the operation type corresponding to the execution at the operation time according to the operation log of the self-service cash register;

[0020] If the operation type is to modify the quantity of goods, update the scanned quantity of goods according to the modification of the quantity of goods, and then judge abnormal operations according to the matching situation between the updated scanned quantity of goods and the settled quantity of goods.

[0021] Wherein, the recognition of the target area for the image frame in the operation video includes:

[0022] Taking the operation console of the self-service cash register as a reference, identify the area where the operation console is located and the area extending a preset distance in the direction of the consumer user from the operation console as the target area.

[0023] Wherein, the consumer user is located in the area extending the preset distance. The method further includes:

[0024] Obtain a behavior recognition model, and use the operation video as an input to obtain a third recognition result output by the behavior recognition model. The third recognition result indicates whether a suspicious limb behavior is detected in the operation video. The suspicious limb behavior is used to represent an abnormal action of the consumer user during the process of scanning and settling goods.

[0025] The judgment of abnormal operations on the consumer user according to the matching situation between the scanned quantity of goods and the settled quantity of goods includes: judging abnormal operations on the consumer user according to the matching situation and the third recognition result.

[0026] Wherein, the operation video is the operation video corresponding to a transaction order that has been paid within a preset time period by the self-service cash register.

[0027] Wherein, the obtaining of the operation video of the consumer user during the process of scanning and settling goods by the self-service cash register includes:

[0028] Identify risk orders that may have abnormal operation behaviors from the transaction orders that have been paid;

[0029] Obtain the operation video corresponding to the risk order and perform abnormal operation recognition.

[0030] A method for constructing a commodity recognition model, the method comprising:

[0031] Obtain a sample image and corresponding annotation information, where the sample image contains image content of a sample user holding a sample commodity, and the annotation information is used to represent the annotation position of the sample commodity in the sample image;

[0032] Obtain an initial model for commodity position recognition;

[0033] Use the sample image as the input of the initial model to obtain the predicted position of the sample commodity in the sample image output by the initial model;

[0034] Train the initial model according to the matching degree between the predicted position and the annotation position of the sample image to obtain a commodity recognition model.

[0035] Wherein, if the position of the sample commodity in the sample image is determined by means of framing with a wireframe of a preset shape, the annotation position and the predicted position are respectively embodied as a set of coordinate values for drawing the boundary of the wireframe, and then a set of coordinate values representing the predicted position and a set of coordinate values representing the annotation position are respectively compared correspondingly to obtain the matching degree.

[0036] A method for constructing a behavior recognition model, the method comprising:

[0037] Obtain a first sample video with suspicious limb behaviors, where the suspicious limb behaviors are used to represent abnormal actions that occur during the process of a sample user performing commodity scanning and settlement through a self-service cash register;

[0038] Obtain an initial model for suspicious limb behavior recognition;

[0039] Use the first sample video as the input of the initial model and train the initial model to obtain a behavior recognition model with the ability to detect suspicious limb behaviors.

[0040] Wherein, the method further comprises:

[0041] Obtain a second sample video without suspicious limb behaviors, and configure the sample quantities of the first sample video and the second sample video according to the ratio of the number of abnormal transaction orders and the number of normal transaction orders generated within a specified time period; the abnormal transaction order is an order with abnormal operation behaviors, and the normal transaction order is an order without abnormal operation behaviors;

[0042] Using the first sample video as the input of the initial model to perform model training on the initial model includes: using the first sample video and the second sample video as the input of the initial model to perform model training on the initial model.

[0043] An abnormal operation recognition device, the device includes:

[0044] An operation video acquisition unit, configured to acquire an operation video of a consumer user during the process of scanning and settling goods through a self-service cash register device;

[0045] A target area recognition unit, configured to perform target area recognition on an image frame in the operation video, where the target area includes an image area related to the consumer user's operation of scanning and settling goods;

[0046] A first result acquisition unit, configured to acquire a commodity recognition model, and use the image frame in which the target area is recognized as the input to obtain a first recognition result output by the commodity recognition model, where the first recognition result includes the position information of the goods to be settled in the image frame when the consumer user holds the goods to be settled and performs a scanning operation;

[0047] A feature information processing unit, configured to extract the commodity feature information of the goods to be settled from the image frame according to the position information, and then perform commodity deduplication processing according to the commodity feature information to obtain the number of scanned commodities of the consumer user;

[0048] An abnormal operation judgment unit, configured to determine the number of settled commodities in the transaction order according to the transaction order corresponding to the operation video, and perform an abnormal operation judgment on the consumer user according to the matching situation between the number of scanned commodities and the number of settled commodities.

[0049] A commodity recognition model construction device, the device includes:

[0050] A sample information acquisition unit, configured to acquire a sample image and corresponding annotation information, where the sample image includes image content of a sample user holding a sample commodity, and the annotation information is used to represent the annotation position of the sample commodity in the sample image;

[0051] An initial model acquisition unit, configured to acquire an initial model for performing commodity position recognition;

[0052] A predicted position acquisition unit, configured to use the sample image as the input of the initial model to obtain the predicted position of the sample commodity in the sample image output by the initial model;

[0053] An initial model training unit, configured to train the initial model according to the matching degree between the predicted position and the labeled position of the sample image, so as to obtain a commodity recognition model.

[0054] A behavior recognition model construction device, the device includes:

[0055] A sample video acquisition unit, configured to acquire a first sample video with suspicious limb behaviors, where the suspicious limb behaviors are used to represent abnormal actions that occur during the process of a sample user performing commodity scanning and settlement through a self-service cash register device;

[0056] An initial model acquisition unit, configured to acquire an initial model for recognizing suspicious limb behaviors;

[0057] An initial model training unit, configured to use the first sample video as the input of the initial model, and train the initial model to obtain a behavior recognition model with the ability to detect suspicious limb behaviors.

[0058] A computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of the method described in any one of the foregoing are implemented.

[0059] An electronic device, including:

[0060] One or more processors; and

[0061] A memory associated with the one or more processors, where the memory is used to store program instructions, and when the program instructions are read and executed by the one or more processors, the steps of the method described in any one of the foregoing are executed.

[0062] A computer program product, including a computer program / computer executable instructions, and when the computer program / computer executable instructions are executed by a processor in an electronic device, the steps of the method described in any one of the foregoing are implemented.

[0063] According to the specific embodiments provided in this application, the following technical effects are disclosed in this application:

[0064] In the embodiments of this application, it is possible to perform video analysis on the operation video of a consumer user performing scanning and settlement through a self-service cash register device, identify the target area, extract commodity feature information from the image content of the commodity held by the user in the target area, and perform commodity deduplication according to the commodity feature information, so as to identify and count the number of scanned commodities for which the user makes a scanning action in the operation video, and then combine the number of settled commodities that have completed payment in the transaction order corresponding to the operation video to perform quantity matching, and determine the abnormal operation of the consumer user according to the matching situation.

[0065] Such a solution can simply and conveniently identify abnormal operations during the product scanning process, without the need for store personnel to participate or equipment modification. It ensures that the store's labor costs and equipment costs are not affected by the abnormal operation identification function, and does not restrict the scanning methods used by users when scanning products, which helps to simplify user operations and improve the user experience.

[0066] Of course, it is not necessary for any product implementing this application to achieve all the above-mentioned advantages simultaneously. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0068] Figure 1 is a flowchart of the abnormal operation identification method provided by the embodiment of the present application;

[0069] Figure 2 is a schematic diagram of the abnormal operation identification device provided by the embodiment of the present application;

[0070] Figure 3 is a schematic diagram of the product identification model construction device provided by the embodiment of the present application;

[0071] Figure 4 is a schematic diagram of the behavior identification model construction device provided by the embodiment of the present application;

[0072] Figure 5 is a schematic diagram of the electronic device provided by the embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0073] The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, rather than all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art belong to the scope of protection of the present application.

[0074] In order to avoid intentional or unintentional abnormal operations such as missed scanning during the self-checkout process, offline physical stores either arrange full-time personnel to conduct real-time inspections in the self-checkout area, or confirm the goods scanned by users in a weighing manner, or verify the scanning behavior of users by analyzing the cross-zone scanning actions from the non-scanned goods area to the scanned goods area through video analysis. These solutions will not only increase the labor cost and equipment cost of the store, but may also affect the user experience due to factors such as manual inspections and the requirement to use specific scanning methods.

[0075] Correspondingly, the embodiments of the present application provide a new abnormal operation recognition solution, which analyzes the operation video of the consumer user during the process of scanning and settling goods through the self-checkout device, and automatically judges the abnormal operations of the user. It neither requires equipment transformation to ensure that the store cost is not affected by the upgrade of the abnormal operation recognition function, nor restricts the scanning method used by the user, which helps to simplify the user operation and improve the user experience.

[0076] The following explains the implementation process of the abnormal operation recognition method in the embodiments of the present application with specific examples. Refer to Figure 1 the flowchart shown, which may include:

[0077] S101: Obtain the operation video of the consumer user during the process of scanning and settling goods through the self-checkout device.

[0078] Generally, after a consumer user selects goods in an offline store, they can scan the product barcodes through the self-checkout device for self-checkout. The camera installed on the self-checkout device can collect the complete operation process of the user's scanning and settlement to obtain the operation video of the embodiments of the present application. Among them, there are various installation positions for the camera. For example, it can be installed on the top of the self-checkout device to collect the operation process of the consumer user in a top-down shooting manner.

[0079] In the embodiments of the present application, based on the operation video, the number of scanned goods of the user during the entire operation process can be analyzed, and then combined with the number of settled goods finally settled and paid, and according to the matching situation between the two, it can be identified whether there are abnormal operation behaviors such as missed scanning during the entire operation process.

[0080] In one example, during the process of the consumer user performing the scanning and settlement operation, the corresponding operation video can be obtained in real time, and then the number of scanned goods can be analyzed in real time, and the number of settled goods actually settled in this operation can be obtained after the user clicks to settle and complete the payment.

[0081] In another example, after the consumer user completes the payment, the operation video recording the user's scanning and settlement operation process can be obtained, and a lag analysis can be performed. According to the number of scanned goods of the user in the video and the number of settled goods in the corresponding transaction order, an abnormal operation judgment can be made.

[0082] In this example, the self-checkout device can perform video analysis on the operation videos collected by this device to complete the determination of abnormal operations. Correspondingly, when the self-checkout device is in an idle state, the operation videos corresponding to the transaction orders that have completed payment and were collected by this device within a preset time period can be obtained. Alternatively, other devices that can communicate with the self-checkout device can perform the determination of abnormal operations. Correspondingly, the other devices can obtain, as appropriate, the operation videos corresponding to the transaction orders that have completed payment and were collected by all the self-checkout devices deployed in the store within a preset time period. As an example, the preset time period can be one day, that is, abnormal operation identification is performed on the transaction orders generated by the store every day to determine the potentially existing abnormal transaction orders.

[0083] Preferably, in an example of performing lag analysis on the operation videos, risk orders that may have abnormal operation behaviors can be first screened out from the transaction orders that have completed payment, and then the operation videos corresponding to this part of the risk orders can be obtained for abnormal operation identification. That is to say, a preliminary determination of abnormal operations can be first performed in the way of risk order investigation. For normal transaction orders, there will be no abnormal operation behaviors, so there is no need to perform the abnormal operation identification processing procedure of the embodiment of the present application on them; for risk orders, there may be abnormal operation behaviors, so more accurate determination of abnormal operations can be performed through the solution of the embodiment of the present application, and finally abnormal transaction orders with abnormal operation behaviors can be identified from the risk orders. Performing abnormal operation identification processing on the screened risk orders helps to improve the overall identification efficiency. Especially when the number of transaction orders in the store is very large and the proportion of abnormal transaction orders is usually small compared with normal transaction orders, the overall identification efficiency of the solution can be significantly improved through risk order investigation.

[0084] The embodiment of the present application can perform risk order investigation in various ways, which will be illustrated by examples below.

[0085] In one way, data calibration can be performed on consumer users to identify users who have had abnormal operation behaviors before. The transaction orders generated by these risk users operating the self-checkout device can be identified as risk orders.

[0086] In another approach, since it is a lagging video analysis, more information can be integrated to screen for risky orders. For example, the operation logs of the self-checkout device can be obtained, which store the types of operations performed at different time points. The types of operations can include adding scanned items, deleting scanned items, modifying the quantity of items, etc. Considering that in practical applications, in addition to missed scans, there may also be abnormal operation behaviors such as deleting items after scanning and maliciously reducing the quantity of items for settlement, this approach can also identify transaction orders that have performed operations such as deleting scanned items and modifying the quantity of items (especially reducing the quantity of items) as risky orders.

[0087] In yet another approach, considering that users with abnormal operation behaviors usually act suspiciously due to guilt, such as looking around, frequently checking their phones, operating the checkout device multiple times, or having long pauses in their actions, the embodiments of this application can also analyze the operation video corresponding to the transaction order and identify the transaction order with suspicious body behaviors as a risky order.

[0088] S102: Identify the target area for the image frames in the operation video. The target area includes the image area related to the consumer user performing the operation of scanning and settling goods.

[0089] For the image frames extracted from the operation video, rough segmentation can be performed first to segment out the target area from the image frame. This target area includes the operation space for the user to perform the operation of scanning and settling goods. In this way, if the image frame contains image content related to the user's operation behavior, then the selected target area will necessarily contain this part of the image content. Through rough segmentation, the amount of image processing can be effectively reduced, which helps to improve the overall recognition efficiency. Subsequently, targeted fine segmentation can be performed based on the target area to segment out the user's hand area from the target area, and then obtain the position of the goods when the user holds the goods to be settled and scans them, providing a technical basis for counting the quantity of scanned goods in this application.

[0090] As an example, taking the operation table of the self-checkout device (which can also be called a placement table, etc.) as a reference, the area where the operation table is located and the area extending a preset distance in the direction of the consumer user from the operation table can be identified as the target area of the embodiments of this application. In practical applications, users may not place all the selected goods on the operation table. For example, they may hold the goods in their hands or place the goods in the shopping cart, etc. Therefore, the area extending the preset distance is also determined as the target area to ensure that the image content related to the user's operation behavior can be accurately and comprehensively identified. Among them, the preset distance can be flexibly set according to usage requirements, and the embodiments of this application do not limit this.

[0091] In addition, due to factors such as the installation position and shooting angle of the camera, distortion may occur when collecting the operation video. After obtaining the operation video in the embodiments of the present application, orthodontic processing may be performed on the operation video first, and then the image frames of the embodiments of the present application may be obtained in a frame extraction sampling manner. The specific process can refer to the related art and will not be elaborated here.

[0092] S103: Obtain a commodity recognition model, and use the image frame that identifies the target area as input to obtain a first recognition result output by the commodity recognition model. The first recognition result includes the position information of the commodity to be settled in the image frame when the consumer user holds the commodity to be settled and performs a barcode scanning operation.

[0093] In the embodiments of the present application, the position information of the commodity in the image frame when the user holds the commodity to be settled and performs barcode scanning can be determined from the target area through a commodity recognition model.

[0094] As an example, the edge contour of the commodity to be settled can be recognized, and accordingly, the position where the commodity to be settled is located can be determined from the target area. Correspondingly, the position information can be embodied as a set of coordinate values that can draw the edge contour of the commodity to be settled.

[0095] As another example, in order to reduce the difficulty and calculation amount of the model to determine the position information, a wireframe with a preset shape can be used to frame out the position where the commodity to be settled is located from the target area. Correspondingly, the position information can be embodied as a set of coordinate values that can draw the boundary of the wireframe. For example, the wireframe with a preset shape is a rectangular frame, and the position information can correspondingly be embodied as the coordinate values of the 4 vertices of the rectangular frame. In practical applications, the position of the commodity to be settled in the target area can also be marked by wireframes of other shapes according to the usage requirements, and the embodiments of the present application do not limit this.

[0096] In the embodiments of the present application, the commodity recognition model can be pre-trained, and when needed, the position information of the commodity held by the user can be determined by the model from the input image frame through model calling.

[0097] Specifically, the construction process of the commodity recognition model can be embodied as:

[0098] Obtain a sample image and corresponding annotation information. The sample image contains the image content of a sample user holding a sample commodity, and the annotation information is used to represent the annotation position of the sample commodity in the sample image; obtain an initial model for commodity position recognition; use the sample image as the input of the initial model to obtain the predicted position of the sample commodity in the sample image output by the initial model; and perform model training on the initial model according to the matching degree between the predicted position and the annotation position of the sample image to obtain a commodity recognition model.

[0099] Among them, the initial model can be embodied as a large model obtained through pre-training technology. Based on the pre-trained large model, model fine-tuning can be performed to obtain the commodity recognition model of the present application. It can be understood that a large model trained on a large scale and broad data can possess various basic capabilities, which can not only improve the expression ability of the large model but also optimize the generalization ability of the large model. Then, based on the fine-tuning technology, the large model can be generalized to different downstream tasks with a small number of training samples to obtain a fine-tuned model corresponding to the downstream task. In this example, the downstream task is the commodity position calibration task, and the fine-tuned model corresponding to the downstream task is the commodity recognition model obtained by model training.

[0100] In the embodiments of the present application, a small number of sample images can be obtained, and position information annotation can be performed on different sample images respectively to obtain the annotation information corresponding to each sample image. For example, the position of the sample commodity in the sample image can be determined by using a wireframe of a preset shape and framing it. The annotation position can be embodied as a set of coordinate values for drawing the boundary of the wireframe. Still taking a rectangular wireframe as an example, the annotation position can be correspondingly embodied as the coordinate values of the 4 vertices of the rectangular wireframe. Correspondingly, the predicted position output by the initial model can also be embodied as a set of coordinate values for drawing the boundary of the wireframe. In this way, a set of coordinate values representing the predicted position and a set of coordinate values representing the annotation position can be used to perform corresponding comparisons respectively to obtain the matching degree between the predicted position and the annotation position of the sample image. Then, when the matching degree meets the requirements of the corresponding loss function, it is determined that the model training is completed, and the commodity recognition model of the embodiments of the present application is obtained.

[0101] As an example, the commodity recognition model of the embodiments of the present application can be a multi-modal model. Specifically, in addition to taking the image frame of the recognized target area as the model input, a preset text can also be input. For example, the preset text is "Detect the commodity in the person's hand". The commodity recognition model can recognize the semantic information of the text content and accordingly recognize the position information of the commodity to be settled from the input image frame for model output.

[0102] In addition, it should be noted that in the embodiments of the present application, the image frames used as model inputs can be all the image frames that have completed target area recognition in the operation video, or can be some of the image frames that have completed target area recognition in the operation video. This is because during the entire operation process, in addition to the operation of holding the commodity to scan the commodity barcode at the scanning port of the self-service checkout device, the user may also perform other operation actions. For example, the user may also perform operations such as swiping and clicking through the device screen; and / or, view a mobile device such as a mobile phone carried with them. Especially when the user holds the mobile phone in their hand, it is very similar to the image content of holding a commodity; and / or, the user has nothing in their hand, etc. For counting the number of scanned commodities, the image frames generated when the user performs other operation actions can be regarded as invalid frames because these image frames do not contain the image content of the user holding a commodity. Therefore, image frame classification can be performed in advance to remove the invalid frames and retain the valid frames containing the image content of the user holding a commodity as the model input and provide it to the commodity recognition model. In this way, the model processing volume can be effectively reduced, and the model recognition efficiency and accuracy can be improved.

[0103] In the embodiments of the present application, valid image frames can be identified and invalid image frames can be removed in various ways. Considering that in actual applications, if the camera captures the operation video from a top-down perspective, for two-dimensional image frames, when the commodity and the user's hand are in a relative position relationship of up and down in the top-down perspective, the situation where the user has nothing in their hand may also be misidentified as the situation of holding a commodity due to the visual error of the top-down shooting, thereby affecting the recognition accuracy of the valid image frames.

[0104] Correspondingly, the embodiments of the present application provide a solution for determining valid image frames through hand detection and gesture recognition, which may specifically include:

[0105] Obtain a gesture recognition model, and use the image frames that have recognized the target area as inputs to obtain the second recognition result output by the gesture recognition model. The second recognition result is used to indicate whether a gesture of holding a commodity is detected in the image frame. The gesture is recognized by the gesture recognition model by marking the key points of the hand joints after recognizing the consumer user's hand in the target area; use the image frames in which the gesture of holding a commodity is detected as the inputs of the commodity recognition model to obtain the first recognition result output by the commodity recognition model correspondingly.

[0106] That is to say, the present application can analyze the relative positional relationship between multiple key points determined by hand joints when a human hand makes different operation actions, and accordingly perform gesture recognition to determine the operation actions to be executed under different gestures, so as to realize the recognition and classification of operation actions such as holding a commodity, holding a mobile phone, having an empty hand, and operating a screen. On the one hand, invalid frames can be filtered out, and the image frames containing the image content of the held commodity are retained and provided to the commodity recognition model; on the other hand, more preferred solutions can be provided by using the gesture classification results. For example, the number of scanned commodities can be updated by swiping a gesture on the operation device screen. For details, please refer to the following introduction and will not be elaborated here for the time being.

[0107] S104: Extract the commodity feature information of the commodity to be settled from the image frame according to the position information, and then perform duplicate removal processing on the commodity according to the commodity feature information to obtain the number of scanned commodities of the consumer user.

[0108] In practical applications, during the process of a user holding a commodity and scanning the barcode through a scanning port, multiple image frames of the commodity are usually obtained. These image frames completely record the trajectory of the commodity moving from the user side to the scanning port side and then returning from the scanning port side to the user side. In order to accurately count the number of scanned commodities, it is necessary to perform duplicate removal processing on these image frames, that is, identify all image frames for the same commodity and correspondingly increase the number of scanned commodities by 1.

[0109] Specifically, after obtaining the position information of the commodity to be settled in the image frame, the commodity feature information can be extracted from the corresponding image frame according to the position information, and then the similarity calculation is performed using the commodity feature information. The commodities to be settled included in multiple image frames with a feature similarity not lower than a preset value are determined to be the same commodity, so as to achieve the purpose of duplicate removal of the commodity.

[0110] In the actual shopping process, a user may purchase multiple identical commodities at the same time. If other commodities are scanned during the scanning process of multiple identical commodities, that is, between any two identical commodities, there is at least one other commodity. The duplicate removal process of the identical commodities can be separated by the other commodities to ensure the accuracy of the recognition of the number of scanned commodities.

[0111] In addition, considering that there will be a certain time interval between the user putting down the held commodity and picking up a new commodity, a preset time interval can also be set, and the identical commodities detected after exceeding the preset time interval are divided into two commodities for duplicate removal processing.

[0112] In this way, by processing each frame of the image frame with position information in the output (including extracting product feature information and removing duplicate products), the number of scanned products by the consumer user in the operation video can be counted, and this number specifically refers to the number of products for which the user has performed the scanning action while holding the products, that is, the number of products the user holds and makes the scanning action. In practical applications, in addition to increasing the number of products entered into the device through the scanning operation, the user can also increase or decrease the number of products entered through device operations. To improve the accuracy of determining the number of scanned products, the embodiments of the present application can also integrate the output result of the gesture recognition model and provide an update solution for the number of scanned products when it is recognized that the user has performed a sliding gesture for operating the device.

[0113] That is to say, the gesture recognition model can also recognize the sliding gesture of the user operating the self-checkout device. Correspondingly, the second recognition result is also used to indicate whether a sliding gesture for operating the self-checkout device is detected in the image frame. The sliding gesture is recognized by the gesture recognition model by annotating the key points of the human hand joints. The method further includes:

[0114] Determine the operation time when the consumer user performs the sliding gesture, and obtain the operation type corresponding to the operation time according to the operation log of the self-checkout device; if the operation type is modifying the number of products, update the number of scanned products according to the modification of the number of products, and determine whether there is an abnormal operation according to the matching situation between the updated number of scanned products and the number of settled products.

[0115] When the present application recognizes the sliding gesture of the user operating the device screen, it can determine the operation type corresponding to the sliding gesture from the operation log according to the operation time of performing the sliding gesture. Then, when the operation type is modifying the number of products, update the number of scanned products according to the modification of the number of products.

[0116] Among them, modifying the number of products can be reflected as increasing the number of products. For example, when purchasing multiple pieces of the same product, the user can scan the barcode of one of the products and complete the addition of multiple products by modifying the number of products. In this case, 1 scanned product can be recognized through image frame analysis, so the number of scanned products +1. If a sliding gesture is recognized afterwards and the operation log indicates that the sliding gesture is an operation for increasing the number of products, the increased number of products can be combined. For example, 2 pieces of the same product are added, and it can be determined that the updated number of scanned products is actually the number of scanned products +1+2, that is, 3 new products are actually scanned this time.

[0117] Alternatively, modifying the quantity of goods can be reflected as reducing the quantity of goods. For example, if a certain item is scanned repeatedly, or multiple pieces of the same item are purchased and some of them are abandoned after all are scanned, the quantity of goods for settlement can be reduced by modifying the quantity of goods. For instance, after counting the quantity of scanned goods, a swiping gesture is recognized, and the operation log indicates that the swiping gesture is an operation to reduce the quantity of goods. Then, in combination with reducing the quantity of goods, such as reducing 1 piece of the same item, it can be determined that the actual updated quantity of scanned goods is the quantity of scanned goods - 1.

[0118] The above quantity update process can, on the one hand, ensure the accuracy of the quantity of scanned goods obtained by statistics, thereby improving the accuracy of the abnormal operation judgment of this application; on the other hand, it also helps to discover more types of abnormal operation behaviors. For example, after the goods are scanned normally, maliciously reducing the quantity of goods or even deleting the goods. That is to say, the abnormal operation recognition of this application is not limited to identifying the behavior of missed scanning of goods, but can also identify operations such as deleting goods and reducing goods that are intended to reduce the order amount.

[0119] S105: Determine the quantity of goods for settlement in the transaction order corresponding to the operation video, and make an abnormal operation judgment on the consumer user according to the matching situation between the quantity of scanned goods and the quantity of goods for settlement.

[0120] On the one hand, the operation video can be analyzed to count the quantity of scanned goods by the user in the operation video; on the other hand, the quantity of goods for settlement actually paid for the transaction order corresponding to the operation video can be obtained from the order information of the transaction order; then, the quantity of scanned goods and the quantity of goods for settlement are matched in quantity, and an abnormal operation judgment is made according to the matching situation.

[0121] In one case, making an abnormal operation judgment according to the matching situation can be reflected as, when the quantities of the two are completely matched, it is determined that the consumer user has no abnormal operation behavior; otherwise, it is determined that there is an abnormal operation behavior, and the transaction order can also be identified as an abnormal transaction order.

[0122] In another case, making an abnormal operation judgment according to the matching situation can be reflected as, when the deviation between the quantities of the two does not exceed a preset value, it is determined that the consumer user has no abnormal operation behavior; otherwise, it is determined that there is an abnormal operation behavior, and the transaction order can also be identified as an abnormal transaction order. Among them, the preset value can be flexibly set according to actual needs, and the embodiments of this application do not make any limitations on this. The preset value can provide a certain tolerance space for the solution of this application to prevent misjudgment.

[0123] In addition, as introduced above, users with abnormal operation behaviors usually act suspiciously due to a guilty conscience, such as looking around, frequently checking their mobile phones, operating the cash register device multiple times, and pausing for a long time. The embodiments of the present application can also perform video analysis to identify whether there are suspicious limb behaviors of consumer users in the operation video, and then make an abnormal operation judgment based on the quantity matching situation.

[0124] Specifically, when performing target area recognition, if the consumer user is within the area at an extended preset distance, the present application can also obtain a behavior recognition model, use the operation video as input, and obtain a third recognition result output by the behavior recognition model. The third recognition result indicates whether suspicious limb behaviors are detected in the operation video, and the suspicious limb behaviors are used to represent abnormal actions of the consumer user during the commodity scanning and settlement process. In this way, an abnormal operation judgment can also be made on the consumer user according to the matching situation and the third recognition result.

[0125] In one case, it can be determined that the consumer user has no abnormal operation behavior when the quantity of scanned commodities and the quantity of settled commodities are exactly matched or the quantity deviation does not exceed a preset value, and the third recognition result indicates that no suspicious limb behaviors are detected.

[0126] In another case, the third recognition result can be used as an auxiliary judgment factor. For example, when the quantity deviation between the quantity of scanned commodities and the quantity of settled commodities is not less than a first value, it is determined that the consumer user has an abnormal operation behavior; when the quantity deviation is lower than a second value, it is determined that the consumer user has no abnormal operation behavior; when the quantity deviation is between the first value and the second value, if the third recognition result indicates that suspicious limb behaviors are detected in the operation video, it can be determined that the consumer user has an abnormal operation behavior, and if the third recognition result indicates that no suspicious limb behaviors are detected in the operation video, it can be determined that the consumer user has no abnormal operation behavior. Here, the first value is greater than the second value.

[0127] The above are only some examples of making an abnormal operation judgment based on the quantity matching situation, the comprehensive quantity matching situation, and the third recognition result. In actual applications, a specific judgment scheme can be determined according to the usage requirements, and the embodiments of the present application do not limit this.

[0128] In the embodiments of the present application, a behavior recognition model can be pre-trained and, when needed, the model can be called to identify suspicious limb behaviors from the input operation video.

[0129] Specifically, the construction process of the behavior recognition model can be embodied as:

[0130] Obtain a first sample video with suspicious limb behaviors, where the suspicious limb behaviors are used to represent abnormal actions that occur during the process of a sample user scanning and settling goods through a self-checkout device; obtain an initial model for identifying suspicious limb behaviors; use the first sample video as the input of the initial model, and perform model training on the initial model to obtain a behavior recognition model with the ability to detect suspicious limb behaviors.

[0131] As an example, the initial model can be embodied as a large model obtained through pre-training technology. Based on the pre-trained large model, model fine-tuning can be performed to obtain the behavior recognition model of the present application. Specifically, some risk videos with suspicious limb behaviors can be labeled and provided to the initial model for learning as the first sample video. Correspondingly, the output result of the initial model can be obtained and compared with the labeled content of the risk video. When the matching degree between the two meets the requirements of the corresponding loss function, it is determined that the model training is completed, and the behavior recognition model of the embodiment of the present application is obtained. Among them, the labeled content and the output result can be embodied as whether there are suspicious limb behaviors in the video, or can be embodied as specific categories of suspicious limb behaviors, such as looking around, frequently checking the mobile phone, operating the checkout device multiple times, and long pauses in actions in the above examples. Specifically, it can be determined according to the usage requirements, and the embodiments of the present application do not make limitations in this regard.

[0132] In addition, some normal operation videos without suspicious limb behaviors can also be labeled and provided to the initial model as the second sample video to improve the training accuracy of the behavior recognition model by setting positive and negative samples. Preferably, the sample quantities of the first sample video and the second sample video can be configured according to the ratio of the abnormal transaction order quantity and the normal transaction order quantity generated within a specified time period, so that the ratio of the two sample quantities is consistent with the ratio of the order quantities. In this way, the real situation of abnormal operation behaviors occurring during the daily transaction process can be better simulated, which helps to further improve the training accuracy of the model. Among them, being consistent can be embodied as the same ratio of the two, or the ratio deviation of the two is within a certain range. Specifically, it can be determined according to the usage requirements, and the embodiments of the present application do not make limitations in this regard.

[0133] As an example, the behavior recognition model of the embodiment of the present application can be a multi-modal model. Specifically, the operation video and a preset text can be used as the model input, where the preset text can be "whether there are suspicious actions by the person in the operation video". The behavior recognition model can recognize the semantic information of the text content and, based on this, perform detection and analysis on the input operation video to determine whether there are suspicious limb behaviors by the user performing the scanning operation, and then perform model output.

[0134] In summary, the present application can perform video analysis on the operation video of the consumer user performing the code scanning and settlement process through the self-service cash register device, identify the target area, extract the commodity feature information from the image content of the commodities held by the user within the target area, and perform commodity deduplication according to the commodity feature information, so as to identify and count the number of scanned commodities for which the user makes a code scanning action in the operation video. Furthermore, by combining the number of settled commodities for which payment is completed in the transaction order corresponding to the operation video, quantity matching is performed, and abnormal operation judgment of the consumer user is realized according to the matching situation.

[0135] With such a solution, abnormal operations during the commodity code scanning process can be simply and conveniently identified, without the need for store personnel to participate or for equipment modification, ensuring that the store's labor costs and equipment costs are not affected by the abnormal operation identification function, and not restricting or constraining the code scanning methods used by users when scanning commodities, which helps to simplify user operations and improve the user experience.

[0136] In addition, in the solution of performing commodity deduplication through commodity feature information and then identifying and counting the number of scanned commodities, there is no need to specifically classify the commodities in the user's hand, which also helps to reduce the calculation amount of the commodity recognition model and improve the overall recognition efficiency. Of course, if it is necessary to obtain the detailed information of the commodities in the user's hand, such as commodity categories, commodity identifiers, etc., more commodity feature information can be extracted according to the usage requirements, and detailed information recognition can be performed based on this. Specifically, it can be determined in combination with the usage requirements, and the embodiments of the present application do not make limitations in this regard.

[0137] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or fully authorized by all parties. And the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of relevant countries and regions, and corresponding operation entrances are provided for users to choose to authorize or refuse.

[0138] Corresponding to the foregoing method embodiment, the embodiment of the present application further provides an abnormal operation identification device, see Figure 2 , the device may include:

[0139] An operation video acquisition unit 201, configured to acquire an operation video of a consumer user performing commodity code scanning and settlement through a self-service cash register device;

[0140] A target area identification unit 202, configured to perform target area identification on the image frames in the operation video, where the target area includes an image area related to the consumer user performing commodity code scanning and settlement operations;

[0141] The first result acquisition unit 203 is configured to acquire a commodity recognition model, and use the image frame that identifies the target area as input to obtain a first recognition result output by the commodity recognition model. The first recognition result includes the position information of the commodity to be settled in the image frame when the consumer user holds the commodity to be settled and performs a scanning operation.

[0142] The feature information processing unit 204 is configured to extract the commodity feature information of the commodity to be settled from the image frame according to the position information, and then perform duplicate commodity processing according to the commodity feature information to obtain the number of scanned commodities of the consumer user.

[0143] The abnormal operation judgment unit 205 is configured to determine the number of settled commodities in the transaction order according to the transaction order corresponding to the operation video, and perform an abnormal operation judgment on the consumer user according to the matching situation between the number of scanned commodities and the number of settled commodities.

[0144] Wherein, the device further includes:

[0145] The second result acquisition unit is configured to acquire a gesture recognition model, and use the image frame that identifies the target area as input to obtain a second recognition result output by the gesture recognition model. The second recognition result is used to indicate whether a gesture of holding a commodity is detected in the image frame. The gesture is recognized by the gesture recognition model by marking the key points of the human hand joints after recognizing the human hand of the consumer user in the target area.

[0146] The first result acquisition unit may specifically be configured to: use the image frame in which a gesture of holding a commodity is detected as the input of the commodity recognition model, and obtain the first recognition result corresponding to the output of the commodity recognition model.

[0147] Wherein, the second recognition result is further used to indicate whether a sliding gesture of operating the self-service cash register is detected in the image frame. The sliding gesture is recognized by the gesture recognition model by marking the key points of the human hand joints. The device further includes:

[0148] The operation type acquisition unit is configured to determine the operation time when the consumer user performs the sliding gesture, and obtain the operation type corresponding to the operation time according to the operation log of the self-service cash register.

[0149] The commodity quantity update unit is configured to, when the operation type is to modify the commodity quantity, update the scanned commodity quantity according to the commodity quantity modification situation, and then perform an abnormal operation judgment according to the matching situation between the updated scanned commodity quantity and the settled commodity quantity.

[0150] Among them, the target area recognition unit is specifically configured to: based on the operation table of the self-service cash register, recognize the area where the operation table is located and the area extending a preset distance from the operation table towards the consumer user direction as the target area.

[0151] Among them, the consumer user is located in the area extending the preset distance, and the device further includes:

[0152] The third result obtaining unit is configured to obtain a behavior recognition model, and use the operation video as an input to obtain a third recognition result output by the behavior recognition model, where the third recognition result indicates whether a suspicious limb behavior is detected in the operation video, and the suspicious limb behavior is used to represent an abnormal action of the consumer user during the commodity scanning and settlement process;

[0153] The abnormal operation judgment unit is specifically configured to: perform an abnormal operation judgment on the consumer user according to the matching situation and the third recognition result.

[0154] Among them, the operation video is the operation video corresponding to a transaction order that has been paid within a preset time period collected by the self-service cash register.

[0155] Among them, the operation video obtaining unit is specifically configured to: identify a risk order that may have an abnormal operation behavior from the transaction order that has been paid; obtain the operation video corresponding to the risk order for abnormal operation recognition.

[0156] Corresponding to the foregoing method embodiment, an embodiment of the present application further provides a commodity recognition model construction device. Refer to Figure 3 , and the device may include:

[0157] The sample information obtaining unit 301 is configured to obtain a sample image and corresponding annotation information, where the sample image includes image content of a sample user holding a sample commodity, and the annotation information is used to represent the annotation position of the sample commodity in the sample image;

[0158] The initial model obtaining unit 302 is configured to obtain an initial model for commodity position recognition;

[0159] The predicted position obtaining unit 303 is configured to use the sample image as an input to the initial model to obtain the predicted position of the sample commodity in the sample image output by the initial model;

[0160] The initial model training unit 304 is configured to perform model training on the initial model according to the matching degree between the predicted position and the annotation position of the sample image to obtain a commodity recognition model.

[0161] Among them, if the position of the sample commodity in the sample image is determined by enclosing it with a wireframe of a preset shape, the marked position and the predicted position are respectively embodied as a set of coordinate values for drawing the boundary of the wireframe. Then, by using a set of coordinate values representing the predicted position and a set of coordinate values representing the marked position, corresponding comparisons are respectively made to obtain the matching degree.

[0162] Corresponding to the foregoing method embodiments, an embodiment of the present application further provides a device for constructing a behavior recognition model. Refer to Figure 4 This device may include:

[0163] A sample video acquisition unit 401, configured to acquire a first sample video with a suspicious limb behavior, where the suspicious limb behavior is used to represent an abnormal action that occurs during the process of a sample user performing commodity code scanning and settlement through a self-service cash register;

[0164] An initial model acquisition unit 402, configured to acquire an initial model for performing suspicious limb behavior recognition;

[0165] An initial model training unit 403, configured to use the first sample video as an input to the initial model, and perform model training on the initial model to obtain a behavior recognition model with the ability to detect suspicious limb behaviors.

[0166] Among them, the sample video acquisition unit is further configured to: acquire a second sample video without a suspicious limb behavior, and configure the sample quantities of the first sample video and the second sample video according to the ratio of the number of abnormal transaction orders and the number of normal transaction orders generated within a specified time period; the abnormal transaction order is an order with an abnormal operation behavior, and the normal transaction order is an order without an abnormal operation behavior;

[0167] The initial model training unit is specifically configured to: use the first sample video and the second sample video as inputs to the initial model, and perform model training on the initial model.

[0168] In addition, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of the method described in any one of the foregoing method embodiments are implemented.

[0169] And an electronic device, including:

[0170] One or more processors; and

[0171] A memory associated with the one or more processors, the memory being configured to store program instructions that, when read and executed by the one or more processors, perform the steps of the method according to any one of the foregoing method embodiments.

[0172] In addition, an embodiment of the present application further provides a computer program product, including a computer program / computer executable instructions, which when executed by a processor in an electronic device, implement the steps of the method according to any one of the foregoing method embodiments.

[0173] Wherein, Figure 5 An exemplary architecture of an electronic device is shown. For example, device 500 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, an aircraft, etc.

[0174] Referring to Figure 5 , device 500 may include one or more of the following components: a processing component 502, a memory 504, a power component 506, a multimedia component 508, an audio component 510, an input / output (I / O) interface 512, a sensor component 514, and a communication component 516.

[0175] The processing component 502 generally controls the overall operation of device 500, such as operations associated with display, telephone calls, data communication, camera operations, and recording operations. The processing component 502 may include one or more processors 520 to execute instructions to complete all or part of the steps of the method provided by the technical solution of the present disclosure. In addition, the processing component 502 may include one or more modules to facilitate the interaction between the processing component 502 and other components. For example, the processing component 502 may include a multimedia module to facilitate the interaction between the multimedia component 508 and the processing component 502.

[0176] The memory 504 is configured to store various types of data to support the operation of device 500. Examples of such data include instructions for any application or method operating on device 500, contact data, phone book data, messages, pictures, videos, etc. The memory 504 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disk.

[0177] The power supply component 506 provides power for various components of the device 500. The power supply component 506 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 500.

[0178] The multimedia component 508 includes a screen that provides an output interface between the device 500 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can not only sense the boundaries of touch or swipe actions, but also detect the duration and pressure associated with the touch or swipe operations. In some embodiments, the multimedia component 508 includes a front camera and / or a rear camera. When the device 500 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each of the front camera and the rear camera can be a fixed optical lens system or have focal length and optical zoom capabilities.

[0179] The audio component 510 is configured to output and / or input audio signals. For example, the audio component 510 includes a microphone (MIC) that is configured to receive external audio signals when the device 500 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals can be further stored in the memory 504 or transmitted via the communication component 516. In some embodiments, the audio component 510 further includes a speaker for outputting audio signals.

[0180] The input / output (I / O) interface 512 provides an interface between the processing component 502 and a peripheral interface module, and the peripheral interface module can be a keyboard, a click wheel, buttons, etc. These buttons can include, but are not limited to: a home button, a volume button, a power-on button, and a lock button.

[0181] The sensor assembly 514 includes one or more sensors for providing status assessments of various aspects of the device 500. For example, the sensor assembly 514 can detect the on / off state of the device 500, the relative positioning of components, such as the display and keypad of the device 500. The sensor assembly 514 can also detect a change in the position of the device 500 or a component of the device 500, the presence or absence of user contact with the device 500, the orientation or acceleration / deceleration of the device 500, and a change in the temperature of the device 500. The sensor assembly 514 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 514 can also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 514 can also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0182] The communication component 516 is configured to facilitate communication between the device 500 and other devices in a wired or wireless manner. The device 500 can access a wireless network based on communication standards, such as WiFi, or a mobile communication network such as 2G, 5G, 4G / LTE, 5G, etc. In an exemplary embodiment, the communication component 516 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 516 further includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0183] In an exemplary embodiment, the device 500 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components for performing the above-described methods.

[0184] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as the memory 504 including instructions, and the above instructions can be executed by the processor 520 of the device 500 to complete the methods provided by the technical solutions of the present disclosure. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0185] As can be seen from the description of the above embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of this application, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0186] Each embodiment in this specification is described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other. The key point of each embodiment is to illustrate the differences from other embodiments. In particular, for a system or a system embodiment, since it is basically similar to the method embodiment, the description is relatively simple. For the relevant parts, reference can be made to the partial description of the method embodiment. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0187] The above provides a detailed introduction to the solution provided by this application. Specific examples are used in this article to elaborate on the principle and implementation manner of this application. The description of the above embodiments is only used to help understand the method and its core idea of this application; at the same time, for those of ordinary skill in the art, according to the idea of this application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to this application.

Claims

1. An abnormal operation recognition method, characterized in that, The method includes: Obtaining an operation video of the process of a consumer user scanning and settling goods through a self-service cash register device; Performing target area recognition on the image frames in the operation video, where the target area includes the image area related to the consumer user's operation of scanning and settling goods; Obtaining a commodity recognition model, and using the image frames with the recognized target area as input to obtain a first recognition result output by the commodity recognition model. The first recognition result includes the position information of the goods to be settled in the image frames when the consumer user holds the goods to be settled and performs a scanning operation; Extracting the commodity feature information of the goods to be settled from the image frames according to the position information, and then performing duplicate removal processing on the commodities according to the commodity feature information to obtain the number of scanned commodities of the consumer user; Determining the number of settled commodities in the transaction order corresponding to the operation video, and judging the abnormal operation of the consumer user according to the matching situation between the number of scanned commodities and the number of settled commodities; 2. The method according to claim 1, wherein The method further includes: Obtaining a gesture recognition model, and using the image frames with the recognized target area as input to obtain a second recognition result output by the gesture recognition model. The second recognition result is used to indicate whether a gesture of holding goods is detected in the image frames. The gesture is recognized by the gesture recognition model by marking the key points of the human hand joints after recognizing the human hand of the consumer user in the target area; Using the image frames with the detected gesture of holding goods as the input of the commodity recognition model to obtain the first recognition result corresponding to the output of the commodity recognition model; 3. The method according to claim 2, wherein The second recognition result is further used to indicate whether a sliding gesture of operating the self-service cash register device is detected in the image frames. The sliding gesture is recognized by the gesture recognition model by marking the key points of the human hand joints. The method further includes: Determining the operation time when the consumer user performs the sliding gesture, and obtaining the operation type corresponding to the operation time according to the operation log of the self-service cash register device; If the operation type is modifying the number of commodities, updating the number of scanned commodities according to the modification of the number of commodities, and then judging the abnormal operation according to the matching situation between the updated number of scanned commodities and the number of settled commodities; 4. The method according to claim 1, characterized in that, The performing target area recognition on the image frames in the operation video includes: Taking the operation table of the self-service cash register device as a reference, identifying the area where the operation table is located and the area extending a preset distance in the direction of the consumer user from the operation table as the target area; 5. The method according to claim 4, characterized in that, The consumer user is located in the area extending the preset distance. The method further includes: Obtaining a behavior recognition model, and using the operation video as input to obtain a third recognition result output by the behavior recognition model. The third recognition result indicates whether a suspicious limb behavior is detected in the operation video, and the suspicious limb behavior is used to represent an abnormal action of the consumer user during the process of scanning and settling goods; Performing an abnormal operation determination on the consumer user according to the matching situation between the scanned commodity quantity and the settled commodity quantity includes: performing an abnormal operation determination on the consumer user according to the matching situation and the third recognition result.

6. The method according to any one of claims 1 to 5, characterized in that The operation video is the operation video corresponding to the transaction order that has been paid within a preset time period and collected by the self-service cash register device.

7. The method according to claim 6, wherein Obtaining the operation video of the process of the consumer user performing commodity scanning and settlement through the self-service cash register device includes: Identifying a risk order that may have an abnormal operation behavior from the transaction orders that have been paid; Obtaining the operation video corresponding to the risk order and performing abnormal operation identification.

8. A method for constructing a commodity recognition model, characterized in that, The method includes: Obtaining a sample image and corresponding annotation information, the sample image includes the image content of the sample user holding the sample commodity, and the annotation information is used to represent the annotation position of the sample commodity in the sample image; Obtaining an initial model for commodity position recognition; Taking the sample image as the input of the initial model, and obtaining the predicted position of the sample commodity in the sample image output by the initial model; Training the initial model according to the matching degree between the predicted position and the annotation position of the sample image to obtain a commodity recognition model.

9. The method according to claim 8, characterized in that If the position of the sample commodity in the sample image is determined by a wireframe of a preset shape in a wireframe selection manner, the annotation position and the predicted position are respectively represented by a set of coordinate values for drawing the boundary of the wireframe, and then a set of coordinate values representing the predicted position and a set of coordinate values representing the annotation position are respectively compared correspondingly to obtain the matching degree.

10. A method for constructing a behavior recognition model, characterized in that, The method includes: Obtaining a first sample video with suspicious limb behavior, where the suspicious limb behavior is used to represent abnormal actions that occur during the process of the sample user performing commodity scanning and settlement through the self-service cash register device; Obtaining an initial model for suspicious limb behavior recognition; Taking the first sample video as the input of the initial model and training the initial model to obtain a behavior recognition model with the ability to detect suspicious limb behavior.

11. The method according to claim 10, characterized in that, The method further includes: Obtaining a second sample video without suspicious limb behavior, and configuring the sample quantities of the first sample video and the second sample video according to the ratio of the number of abnormal transaction orders and the number of normal transaction orders generated within a specified time period; the abnormal transaction order is an order with an abnormal operation behavior, and the normal transaction order is an order without an abnormal operation behavior; The step of taking the first sample video as the input of the initial model and training the initial model includes: taking the first sample video and the second sample video as the input of the initial model and training the initial model.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 11.

13. An electronic device, characterized in that, Including: One or more processors; And A memory associated with the one or more processors, the memory being configured to store program instructions that, when read and executed by the one or more processors, perform the steps of the method according to any one of claims 1 to 11.

14. A computer program product, comprising a computer program / computer-executable instructions, characterized in that, When the computer program / computer-executable instructions are executed by a processor in an electronic device, the steps of the method according to any one of claims 1 to 11 are implemented.

Images