Reception behavior identification method, device and equipment

Abstract

The embodiment of the invention provides a reception behavior identification method, device and equipment. The method comprises the following steps: acquiring a video stream sent by shooting equipment and an audio stream sent by audio acquisition equipment; the device identifier of the audio acquisition device is associated with the first user identifier; determining a first reception fragment in the video stream, and extracting a first reception feature corresponding to the first reception fragment; determining a second reception fragment in the audio stream, and extracting a second reception feature corresponding to the second reception fragment; and matching the first reception fragment and the second reception fragment according to a target relevancy between the first reception feature and the second reception feature, and determining a target reception fragment and a target user identifier corresponding to each target reception behavior. Therefore, the electronic equipment can realize automatic identification of the target reception behavior based on the video stream and the audio stream, manual registration is not needed, and meanwhile, the reception fragments are matched based on the target relevancy of the reception characteristics, so that the accuracy of reception behavior identification can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a reception behavior recognition method, device and equipment. Background Art

[0002] With the continuous development of the Internet and artificial intelligence technology, merchants (or stores, shops, operators, etc.) in offline retail scenarios usually need to obtain customer flow-related data such as reception behavior data to assist in making operational decisions.

[0003] In related technologies, electronic devices usually identify and obtain staff's reception behavior data based on technologies such as face recognition and indoor positioning, or identify and obtain reception behavior data through manual registration and confirmation by staff. This reception behavior recognition method has low recognition accuracy in complex offline scenarios, and the manual registration method is costly. Summary of the invention

[0004] Multiple aspects of the present application provide a reception behavior recognition method, device and equipment, which can reduce labor costs and improve the accuracy of reception behavior recognition.

[0005] In a first aspect, an embodiment of the present application provides a reception behavior recognition method, comprising:

[0006] Acquire a video stream sent by a shooting device and an audio stream sent by an audio acquisition device; the device identifier of the audio acquisition device is associated with the first user identifier;

[0007] Determining a first reception segment in the video stream, and extracting a first reception feature corresponding to the first reception segment;

[0008] determining a second reception segment in the audio stream, and extracting a second reception feature corresponding to the second reception segment;

[0009] According to the target correlation between the first reception feature and the second reception feature, the first reception segment and the second reception segment are matched to determine the target reception segment and the target user identifier corresponding to each target reception behavior.

[0010] In a possible implementation manner, determining the first reception segment in the video stream includes:

[0011] Determine a first movement trajectory of the first user and a second movement trajectory of the second user in the video stream;

[0012] determining a target user pair according to a distance between the first moving trajectory and the second moving trajectory;

[0013] For each image frame corresponding to the target user pair, if in the image frame, a first orientation angle of a first user and a second orientation angle of a second user in the target user pair meet a preset angle condition, then determining the image frame as a reception image frame including a reception behavior;

[0014] Aggregate the reception image frames of the same target user pair to obtain the first reception segment.

[0015] In a possible implementation manner, extracting the first reception feature corresponding to the first reception segment includes:

[0016] Determine the first reception start and end time corresponding to the first reception segment; and / or,

[0017] Determine, according to a preset action detection model, a first action type and a first action time corresponding to the first reception segment; and / or,

[0018] determining a first customer attribute corresponding to the first reception segment according to the body features of the second user in the first reception segment; and / or,

[0019] determining, according to a preset object detection model, a first object attribute and a first object interaction time corresponding to the first reception segment; and / or,

[0020] According to a preset human body detection model, a first clerk feature corresponding to the first reception segment is determined.

[0021] In a possible implementation manner, the determining the second reception segment in the audio stream includes:

[0022] Performing text conversion processing on the audio stream to obtain text sequence data corresponding to the audio stream;

[0023] Inputting the text sequence data into a preset text recognition model to determine the reception behavior corresponding to the text sequence data and the start and end times of the reception behavior;

[0024] According to the start and end times of the reception behavior, a second reception segment corresponding to each first user in the audio stream is determined.

[0025] In a possible implementation manner, extracting the second reception feature corresponding to the second reception segment includes:

[0026] Determine the second reception start and end time corresponding to the second reception segment; and / or,

[0027] determining, according to a preset text recognition model, a second action type and a second action time corresponding to the second reception segment; and / or,

[0028] determining a second customer attribute corresponding to the second reception segment according to an original audio signal corresponding to the second reception segment and a preset sound recognition model; and / or,

[0029] According to the preset text recognition model, a second object attribute and a second object interaction time corresponding to the second reception segment are determined.

[0030] In a possible implementation, the method further includes:

[0031] Determine the start and end time correlation between the first reception start and end time and the second reception start and end time; and / or,

[0032] determining the action relevance according to the matching result of the first action type and the second action type, the first action time and the second action time; and / or,

[0033] Determining the customer attribute relevance according to the matching result of the first customer attribute and the second customer attribute; and / or,

[0034] Determining the item relevance according to the matching result of the first item attribute and the second item attribute, the first item interaction time and the second item interaction time; and / or,

[0035] Determining the store clerk relevance according to the similarity between the first store clerk feature and the preset store clerk features in the preset database;

[0036] A target correlation between the first reception feature and the second reception feature is determined based on at least one of the start and end time correlation, the action correlation, the customer attribute correlation, the item correlation and the clerk correlation.

[0037] In a possible implementation manner, matching the first reception segment and the second reception segment to determine the target reception segment and the target user identifier corresponding to each target reception behavior includes:

[0038] According to the target correlation between the first reception feature and the second reception feature, determining the first reception segment and the second reception segment having the greatest target correlation with the first reception segment as the target reception segment corresponding to the target reception behavior;

[0039] The first user identifier corresponding to the second reception segment in the target reception segment is determined as the target user identifier.

[0040] In a possible implementation manner, matching the first reception segment and the second reception segment to determine the target reception segment and the target user identifier corresponding to each target reception behavior includes:

[0041] Determine a correlation matrix corresponding to the first reception segment and the second reception segment according to the target correlation between the first reception feature and the second reception feature; the elements in the correlation matrix are the target correlations;

[0042] Deleting the elements whose target relevance is less than a preset relevance threshold from the relevance matrix to obtain an updated relevance matrix;

[0043] With respect to the updated correlation matrix, the first reception segment and the second reception segment are matched according to a preset planning model to determine the target reception segment and the target user identifier corresponding to each target reception behavior.

[0044] In a possible implementation manner, matching the first reception segment and the second reception segment according to the preset planning model to determine the target reception segment and the target user identifier corresponding to each target reception behavior includes:

[0045] Determine the decision variables corresponding to each non-zero element in the updated correlation matrix;

[0046] Determine an objective function and an objective constraint condition corresponding to the objective function according to each of the decision variables and the non-zero elements corresponding to the decision variables;

[0047] Calculating the objective function according to the objective constraint condition to obtain the objective value corresponding to each of the decision variables;

[0048] According to the target value, the target reception segment and the target user identifier corresponding to each target reception behavior are determined.

[0049] In a second aspect, an embodiment of the present application provides a reception behavior recognition device, including:

[0050] An acquisition module, used to acquire a video stream sent by a shooting device and an audio stream sent by an audio acquisition device; the device identifier of the audio acquisition device is associated with the first user identifier;

[0051] A first determining module, configured to determine a first reception segment in the video stream, and extract a first reception feature corresponding to the first reception segment;

[0052] A second determining module, configured to determine a second reception segment in the audio stream, and extract a second reception feature corresponding to the second reception segment;

[0053] The matching module is used to match the first reception segment and the second reception segment according to the target correlation between the first reception feature and the second reception feature, and determine the target reception segment and the target user identifier corresponding to each target reception behavior.

[0054] In a possible implementation manner, the first determining module is specifically configured to:

[0055] Determine a first movement trajectory of the first user and a second movement trajectory of the second user in the video stream;

[0056] determining a target user pair according to a distance between the first moving trajectory and the second moving trajectory;

[0057] For each image frame corresponding to the target user pair, if in the image frame, a first orientation angle of a first user and a second orientation angle of a second user in the target user pair meet a preset angle condition, then determining the image frame as a reception image frame including a reception behavior;

[0058] Aggregate the reception image frames of the same target user pair to obtain the first reception segment.

[0059] In a possible implementation manner, the first determining module is specifically configured to:

[0060] Determine the first reception start and end time corresponding to the first reception segment; and / or,

[0061] Determine, according to a preset action detection model, a first action type and a first action time corresponding to the first reception segment; and / or,

[0062] determining a first customer attribute corresponding to the first reception segment according to the body features of the second user in the first reception segment; and / or,

[0063] determining, according to a preset object detection model, a first object attribute and a first object interaction time corresponding to the first reception segment; and / or,

[0064] According to a preset human body detection model, a first clerk feature corresponding to the first reception segment is determined.

[0065] In a possible implementation manner, the second determining module is specifically configured to:

[0066] Performing text conversion processing on the audio stream to obtain text sequence data corresponding to the audio stream;

[0067] Inputting the text sequence data into a preset text recognition model to determine the reception behavior corresponding to the text sequence data and the start and end times of the reception behavior;

[0068] According to the start and end times of the reception behavior, a second reception segment corresponding to each first user in the audio stream is determined.

[0069] In a possible implementation manner, the second determining module is specifically configured to:

[0070] Determine the second reception start and end time corresponding to the second reception segment; and / or,

[0071] Determine, according to a preset text recognition model, a second action type and a second action time corresponding to the second reception segment; and / or,

[0072] determining a second customer attribute corresponding to the second reception segment according to an original audio signal corresponding to the second reception segment and a preset sound recognition model; and / or,

[0073] According to the preset text recognition model, a second object attribute and a second object interaction time corresponding to the second reception segment are determined.

[0074] In a possible implementation manner, the device is further used for:

[0075] Determine the start and end time correlation between the first reception start and end time and the second reception start and end time; and / or,

[0076] determining the action relevance according to the matching result of the first action type and the second action type, the first action time and the second action time; and / or,

[0077] Determining the customer attribute relevance according to the matching result of the first customer attribute and the second customer attribute; and / or,

[0078] Determining the item relevance according to the matching result of the first item attribute and the second item attribute, the first item interaction time and the second item interaction time; and / or,

[0079] Determining the store clerk relevance according to the similarity between the first store clerk feature and the preset store clerk features in the preset database;

[0080] A target correlation between the first reception feature and the second reception feature is determined based on at least one of the start and end time correlation, the action correlation, the customer attribute correlation, the item correlation and the clerk correlation.

[0081] In a possible implementation manner, the matching module is specifically configured to:

[0082] According to the target correlation between the first reception feature and the second reception feature, determining the first reception segment and the second reception segment having the greatest target correlation with the first reception segment as the target reception segment corresponding to the target reception behavior;

[0083] The first user identifier corresponding to the second reception segment in the target reception segment is determined as the target user identifier.

[0084] In a possible implementation manner, the matching module is specifically configured to:

[0085] Determine a correlation matrix corresponding to the first reception segment and the second reception segment according to the target correlation between the first reception feature and the second reception feature; the elements in the correlation matrix are the target correlations;

[0086] Deleting the elements whose target relevance is less than a preset relevance threshold from the relevance matrix to obtain an updated relevance matrix;

[0087] With respect to the updated correlation matrix, the first reception segment and the second reception segment are matched according to a preset planning model to determine the target reception segment and the target user identifier corresponding to each target reception behavior.

[0088] In a possible implementation manner, the matching module is specifically configured to:

[0089] Determine the decision variables corresponding to each non-zero element in the updated correlation matrix;

[0090] Determine an objective function and an objective constraint condition corresponding to the objective function according to each of the decision variables and the non-zero elements corresponding to the decision variables;

[0091] Calculating the objective function according to the objective constraint condition to obtain the objective value corresponding to each of the decision variables;

[0092] According to the target value, the target reception segment and the target user identifier corresponding to each target reception behavior are determined.

[0093] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory and a processor;

[0094] The memory stores computer-executable instructions;

[0095] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the reception behavior recognition method described in any one of the first aspects.

[0096] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the reception behavior recognition method described in any one of the first aspects.

[0097] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the reception behavior recognition method shown in any one of the first aspects.

[0098] In an embodiment of the present application, a video stream sent by a shooting device and an audio stream sent by an audio acquisition device are obtained; the device identifier of the audio acquisition device is associated with a first user identifier; a first reception segment in the video stream is determined, and a first reception feature corresponding to the first reception segment is extracted; a second reception segment in the audio stream is determined, and a second reception feature corresponding to the second reception segment is extracted; the first reception segment and the second reception segment are matched according to the target correlation between the first reception feature and the second reception feature, and the target reception segment and the target user identifier corresponding to each target reception behavior are determined. In the present application, the electronic device obtains a video stream and an audio stream, extracts the first reception segment from the video stream and determines the first reception feature, extracts the second reception segment from the audio stream and determines the second reception feature, and then matches the first reception segment and the second reception segment according to the target correlation between the reception features, and finally determines the target reception segment and the target user identifier corresponding to the same target reception behavior. In this way, the electronic device can realize automatic recognition of the target reception behavior based on the video stream and the audio stream, without manual registration and confirmation, and at the same time, the reception segment is matched based on the target correlation of the reception feature, which can improve the accuracy of reception behavior recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0099] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0100] Figure 1 A flowchart of a reception behavior identification method provided for an exemplary embodiment of the present application;

[0101] Figure 2 A schematic diagram of a process of determining a first reception segment and extracting a first reception feature provided by an exemplary embodiment of the present application;

[0102] Figure 3 A schematic diagram of a process of determining a second reception segment and extracting a second reception feature provided by an exemplary embodiment of the present application;

[0103] Figure 4 A schematic diagram of a target reception segment and a target user identification determination process provided by an exemplary embodiment of the present application;

[0104] Figure 5 A schematic diagram of a target relevance calculation process provided for an exemplary embodiment of the present application;

[0105] Figure 6 A timing diagram of a first reception segment and a second reception segment provided for an exemplary embodiment of the present application;

[0106] Figure 7 A schematic diagram of association and matching between a first reception segment and a second reception segment provided for an exemplary embodiment of the present application;

[0107] Figure 8 A processing logic diagram of reception behavior recognition provided for an exemplary embodiment of the present application;

[0108] Fig. 9 A schematic diagram of the structure of a reception behavior recognition device provided by an exemplary embodiment of the present application;

[0109] Fig.10 A schematic structural diagram of an electronic device provided for an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0110] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of them. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application. It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0111] In offline retail and other service industries, reception behavior can refer to the interaction between sellers and buyers, product and service providers and providers within retail business premises. For example, the staff in a car sales store receives customers, communicates, and sells. Similar interactive behaviors may also occur in sales offices and clothing, jewelry, beauty, electronics, hardware and other stores, discount stores, wholesale markets, supermarkets, gas stations and other retail venues. Of course, they can also occur in exhibition halls, banks, gyms, medical treatment, education and training and other types of service venues. The embodiments of this application do not limit the specific types of business premises in offline retail and other service industries.

[0112] In offline retail scenarios, the process of staff receiving customers is crucial to the realization of sales goals and service effects. Therefore, operators pay more attention to whether the staff inside the store can receive customers and provide services in a timely manner. At this time, it is necessary to identify and count the reception behavior of the staff, identify and count the reception situations of different staff members, and obtain information such as work saturation and conversion rate. These reception behavior data can be used as reference data for optimizing staff service quality and adjusting operation strategies. That is, in offline retail scenarios, it is necessary to identify reception behaviors and determine the identity of the staff corresponding to each reception behavior.

[0113] In the relevant technologies, electronic devices usually identify and obtain the reception behavior data of staff based on technologies such as face recognition and indoor positioning. Since offline retail scenarios are relatively complex, there may be multiple customers and multiple staff members inside the store, and there may be complex situations such as some customers being received and some not being received, one staff member may successively receive multiple customers, and one customer may also be successively received by multiple staff members. The reception behavior recognition method based on technologies such as face recognition and indoor positioning in the relevant technologies cannot achieve accurate recognition of reception behavior. Face recognition technology cannot accurately identify the identity of the staff member corresponding to each reception behavior in complex scenarios with occlusion. In addition, technologies such as indoor positioning have high requirements for hardware equipment, resulting in high system costs.

[0114] In addition, in the related technology, store operators can also identify and obtain reception behavior data through manual registration and confirmation by staff. This reception behavior identification method increases the workload of staff, and the time cost and labor cost of manual registration are high.

[0115] In order to solve the above problems, the present application provides a reception behavior recognition method, device and equipment, in which the electronic device obtains the video stream sent by the shooting device and the audio stream sent by the audio acquisition device, and extracts the first reception segment from the video stream and determines the first reception feature, extracts the second reception segment from the audio stream and determines the second reception feature, and then matches the first reception segment and the second reception segment according to the target correlation between the first reception feature and the second reception feature, and finally determines the target reception segment and the target user identification corresponding to the same target reception behavior. In this way, the electronic device can realize automatic recognition of the target reception behavior based on the video stream and the audio stream, without manual registration and confirmation, and at the same time, the reception segment is matched based on the target correlation of the reception feature, which can improve the accuracy of reception behavior recognition.

[0116] The technical solution shown in the present application is described in detail below through specific embodiments. It should be noted that the following embodiments can exist independently or in combination with each other, and the same or similar contents will not be described repeatedly in different embodiments.

[0117] Figure 1 A flowchart of a reception behavior recognition method provided by an exemplary embodiment of the present application. Figure 1 , the reception behavior recognition method may include:

[0118] S101, obtaining a video stream sent by a shooting device and an audio stream sent by an audio acquisition device; a device identifier of the audio acquisition device is associated with a first user identifier.

[0119] The execution subject of the embodiment of the present application may be an electronic device, or a reception behavior recognition device provided in the electronic device. The reception behavior recognition device may be implemented by software, or by a combination of software and hardware. For ease of understanding, the following description will be given by taking the execution subject as an electronic device as an example. The electronic device may specifically refer to an edge server, a cloud, a mobile phone, a wearable device, etc., and may of course be other types. The embodiment of the present application does not limit the specific type of the electronic device.

[0120] In the embodiment of the present application, the shooting device may refer to a camera in an offline retail scene, specifically a network camera (Internet Protocol Camera, IPC), a surveillance camera, etc. The number of shooting devices may be one, two or more, and the embodiment of the present application does not limit the specific type and number of shooting devices. The video stream may refer to a video stream shot by a shooting device, which may generally refer to a video stream shot within a preset time period, for example, the preset time period may be from one hour before the start of business to one hour after the end of business. The audio acquisition device may refer to an audio collection device carried by staff in an offline retail scene, specifically a smart work badge, a smart headset, a voice recorder, etc. The audio acquisition device can collect voice information of the staff and the surrounding to form an audio stream, and send the audio stream to the electronic device. The first user may refer to the staff inside the store in the offline retail scene.

[0121] It should be noted that for the second user (i.e., customer, consumer, etc.) in the video stream, the electronic device can detect the appearance position of different second users in each image frame of the video stream based on detection and recognition algorithms such as pedestrian detection, pedestrian tracking and pedestrian re-recognition, for example, a matrix detection frame can be used to identify, and each second user can be associated with a second user identifier.

[0122] (Identity document, ID), to achieve accurate identification and distinction between different second users. However, for staff, since the first user in offline retail scenarios usually wears the same work clothes, the accuracy of traditional detection and recognition algorithms such as pedestrian re-identification is not high, and it is impossible to accurately identify the identity of the first user. In an embodiment of the present application, the audio acquisition device has an association relationship with the first user, that is, the device identifier of the audio acquisition device is associated with the first user identifier, which can specifically be a one-to-one association relationship. The electronic device can determine the first user identifier based on the device identifier corresponding to the audio stream uploaded by the audio acquisition device worn by the first user, thereby accurately identifying the identity of the staff.

[0123] In this step, in an offline retail scenario, the electronic device can obtain the video stream captured and sent by the shooting device, and at the same time obtain the audio stream captured and sent by the audio acquisition device worn by the first user, and can subsequently recognize the reception behavior based on the video stream and audio stream.

[0124] It should be noted that the number of customers in the customer flow system refers to the customer flow data before identity deduplication, the number of customers refers to the customer flow data after identity deduplication, and the customer batch refers to the customer flow data obtained by counting the families, friends and other teams that come to the store at the same time as a whole only once. The reception behavior in the embodiment of the present application does not specifically distinguish between the number of people, the number of people, and the batch. In other words, the reception behavior identification method in the embodiment of the present application is applicable to scenarios that require the number of customers, the number of people, and the batch.

[0125] S102: Determine a first reception segment in a video stream, and extract a first reception feature corresponding to the first reception segment.

[0126] In the embodiment of the present application, the first reception segment may refer to a video segment in which reception behavior exists in the video stream. The first reception feature may refer to a specific reception feature of the first reception segment. For example, the first reception feature may include the first reception start and end time, the first action type and the first action time, the first customer attribute, the first item attribute and the first item interaction time, the first clerk feature, etc. Among them, the first reception start and end time may refer to the start time and end time corresponding to the first reception segment; the first action type and the first action time may refer to the specific reception action of the first user in the first reception segment and the time when the reception action occurs. The first customer attribute may refer to the human attributes of the second user (i.e., customer, consumer, etc.) in the first reception segment, and may specifically include the age group (children, young and middle-aged, elderly, etc.) and gender of the second user. The first item attribute and the first item interaction time may refer to the specific attributes of the item (such as item type, item model, item color, etc.) of the reception behavior interaction in the first reception segment and the time when the item interaction occurs. For example, the item may refer to a car in a car sales scene, etc. The first clerk feature may refer to the appearance feature of the first user in the first reception segment, and may specifically include the human features, head features, or face features of the first user, etc. Of course, the first reception feature corresponding to the first reception segment may also include other types of features, which is not limited in this embodiment of the present application.

[0127] In this step, after acquiring the video stream sent by the shooting device, the electronic device can detect and analyze the video stream to determine the first reception segment included in the video stream. Specifically, the electronic device can detect people in the video stream based on a human body detection model, etc., to determine the first user and the second user in the video stream; then, the first reception segment corresponding to each reception behavior in the video stream can be determined based on the trajectory distance between the first user and the second user and the human body orientation angle. After determining multiple first reception segments in the video stream, the electronic device can extract the first reception features corresponding to each first reception segment.

[0128] S103: Determine a second reception segment in the audio stream, and extract a second reception feature corresponding to the second reception segment.

[0129] In the embodiment of the present application, the second reception segment may refer to an audio segment in which the reception behavior exists in the audio stream. The second reception feature may refer to a specific reception feature of the second reception segment. For example, the second reception feature may include the second reception start and end time, the second action type and the second action time, the second customer attribute, the second item attribute and the second item interaction time, etc. Of course, the second reception feature may also include other types of features, which are not limited in the embodiment of the present application.

[0130] In this step, after acquiring the audio stream sent by the audio acquisition device, the electronic device can detect and analyze the audio stream to determine the second reception segment included in the audio stream. Specifically, the electronic device can convert the audio stream into text sequence data, and then detect and analyze the text sequence data based on a language model or a classification model, etc., to determine that there are multiple second reception segments in the audio stream containing reception behaviors. Subsequently, the electronic device can extract the second reception features corresponding to the second reception segments from each second reception segment.

[0131] S104: Match the first reception segment and the second reception segment according to the target correlation between the first reception feature and the second reception feature, and determine the target reception segment and the target user identifier corresponding to each target reception behavior.

[0132] In the embodiment of the present application, the target relevance may refer to the overall correlation between the first reception feature and the second reception feature, and the target relevance may be comprehensively determined according to the feature relevances corresponding to different feature types between the first reception feature and the second reception feature. The target reception behavior may refer to the same reception behavior corresponding to the matched first reception segment and the second reception segment. The target reception segment may refer to the first reception segment and the second reception segment corresponding to the target reception behavior. The target user identifier may refer to the user identifier of the first user corresponding to the target reception behavior.

[0133] In this step, after the electronic device determines the first reception segment and the first reception feature based on the video stream and determines the second reception segment and the second reception feature based on the audio stream, it can calculate the target correlation between the first reception feature and the second reception feature; then, the first reception segment and the second reception segment are matched in pairs according to the target correlation to determine the target reception segment corresponding to the same target reception behavior, and the target user identifier corresponding to the target reception behavior is determined according to the first user identifier corresponding to the audio stream in the target reception segment, so as to achieve accurate identification of the reception behavior and the identity of the staff, and ensure the accuracy and comprehensiveness of the reception behavior identification.

[0134] In an embodiment of the present application, the electronic device obtains a video stream sent by a shooting device and an audio stream sent by an audio acquisition device; the device identifier of the audio acquisition device corresponds to the first user identifier one by one; the first reception segment in the video stream is determined, and the first reception feature corresponding to the first reception segment is extracted; the second reception segment in the audio stream is determined, and the second reception feature corresponding to the second reception segment is extracted; the first reception segment and the second reception segment are matched according to the target correlation between the first reception feature and the second reception feature, and the target reception segment and the target user identifier corresponding to each target reception behavior are determined. In the present application, the electronic device obtains a video stream and an audio stream, extracts the first reception segment from the video stream and determines the first reception feature, extracts the second reception segment from the audio stream and determines the second reception feature, and then matches the first reception segment and the second reception segment according to the target correlation between the reception features, and finally determines the target reception segment and the target user identifier corresponding to the same target reception behavior. In this way, the electronic device can realize automatic recognition of the target reception behavior based on the video stream and the audio stream, without manual registration and confirmation, and at the same time, the reception segment is matched based on the target correlation of the reception feature, which can improve the accuracy of reception behavior recognition.

[0135] Based on the above embodiments, Figure 2 A flowchart of determining the first reception segment and extracting the first reception feature provided by an exemplary embodiment of the present application. Figure 2 The process of determining the first reception segment and extracting the first reception feature may specifically include:

[0136] S201: Determine a first movement trajectory of a first user and a second movement trajectory of a second user in a video stream.

[0137] In the embodiment of the present application, the first moving trajectory may refer to the moving trajectory of the staff in the video stream. The second moving trajectory may refer to the moving trajectory of the customer in the video stream. The moving trajectory may be composed of a series of trajectory point information, and each trajectory point information may include the coordinate information of the trajectory point and the coordinate time.

[0138] Specifically, after acquiring the video stream sent by the shooting device, the electronic device can first perform human detection on each video frame of the video stream based on a human detection model, such as a real-time target detection algorithm (You Only Look Once, YOLO) and a target detection model (Detection Transformer, DETR), and determine the human detection frame corresponding to each human body in the image frame; then the electronic device can extract the appearance feature vector of the human image in each human detection frame based on the pedestrian re-identification model to obtain the pedestrian re-identification feature of each human body, that is, the human feature; then the electronic device can realize multi-target tracking of each human body according to the similarity of human features between different image frames, the overlap ratio of human detection frames between frames, the distance of human detection frames between frames, and the motion characteristics of human trajectories in historical data, and obtain the corresponding movement trajectory of each human body, and each movement trajectory can correspond to a human trajectory identifier.

[0139] Afterwards, the electronic device can classify the identified human body trajectory using a human body image classification model or an image retrieval algorithm, and determine the first movement trajectory of the first user and the second movement trajectory of the second user. The image classification model is trained using images of human bodies wearing work clothes and a large number of human body images not wearing work clothes, and the image retrieval algorithm calculates the similarity between human body images and a pre-established human body image base in work clothes based on human body features to achieve classification of customers and staff.

[0140] In an embodiment of the present application, for a large number of movement tracks detected in the video streams of multiple shooting devices in a target place (store, shop, supermarket, etc.) within a preset time period (for example, within a day's business hours), the electronic device can use a clustering algorithm to aggregate the movement tracks according to the similarity between human features to obtain a long-term, cross-camera movement track. In this process, in theory, each second user and each first user has a unique user identifier, but because the first user may wear the same work clothes, the pedestrian re-identification model cannot effectively distinguish different staff members in this case, so the first user identifier obtained based on the above video stream detection process is likely to be inaccurate. The electronic device can determine the first movement track belonging to the first user and the second movement track belonging to the second user in the movement track by classifying or retrieving the movement track, that is, the electronic device can determine that a certain movement track belongs to the first user based on the video stream, but cannot accurately determine the first user identifier of the first user, and can be further determined and identified based on the audio stream later.

[0141] It should be noted that the electronic device may also use other algorithms or models to determine the first movement trajectory and the second movement trajectory in the video stream, and the embodiments of the present application are not limited to this.

[0142] S202: Determine a target user pair according to a distance between the first moving trajectory and the second moving trajectory.

[0143] In the embodiment of the present application, the target user pair may refer to a first user and a second user pair that may have reception behavior in the video stream, that is, a customer and a staff member pair. The distance between the first movement trajectory and the second movement trajectory may refer to the distance of the coordinate information at the same time, specifically, the distance between the midpoints of the bottom edge of the human body detection frame, etc.

[0144] In this step, after determining the first moving trajectory and the second moving trajectory in the video stream, the electronic device can match the first moving trajectory with the second moving trajectory in pairs to determine the distance between the first moving trajectory and the second moving trajectory. If the distance is less than a preset distance threshold, the electronic device can form a target user pair with the first user corresponding to the first moving trajectory and the second user corresponding to the second moving trajectory. The preset distance threshold may refer to a preset distance critical value. When the distance between the first moving trajectory and the second moving trajectory is less than the preset distance threshold, the electronic device can determine that the distance between the first user and the second user is relatively close, and there may be reception behavior. The specific value of the preset distance threshold can be flexibly set based on actual needs. For example, it can be 4 times the average width of the human body detection frame of the first user and the second user, etc., and the embodiments of the present application are not limited to this.

[0145] S203. For each image frame corresponding to the target user pair, if in the image frame, a first orientation angle of the first user and a second orientation angle of the second user in the target user pair meet a preset angle condition, then determine that the image frame is a reception image frame including reception behavior.

[0146] In the embodiment of the present application, the first orientation angle may refer to the orientation angle of the first user in the video stream image frame. The second orientation angle may refer to the orientation angle of the second user in the video stream image frame. The preset angle condition may refer to the pre-set reception behavior orientation angle condition. The reception image frame may refer to an image frame with reception behavior.

[0147] In this step, for the target user pairs selected by the electronic device, based on the human body angle estimation model, the first orientation angle of the first user and the second orientation angle of the second user in the target user pair can be determined. The human body angle estimation model can be trained in a variety of different ways. In a possible implementation, the electronic device first obtains training samples, which are a large number of human body images with manually labeled orientation categories. The orientation categories can include 8 categories: front, back, left, right, slightly left of the front, slightly right of the front, slightly left of the back, and slightly right of the back. Then, the electronic device can use these training samples to train a neural network based on the Residual Network (ResNet) structure. Specifically, an 8-class classification model can be trained to obtain the confidence values of the human body images in the 8 orientation categories. The category with the largest confidence value is taken as the human body orientation angle estimation result. By calculating the loss value through the loss function and adjusting the parameters, the human body angle estimation model can be finally obtained through iterative training. Subsequently, in the inference process, based on the human body angle estimation model, the electronic device can determine the orientation angles corresponding to the first user and the second user in the video stream image frame. The orientation angle is specifically one of 8 orientation angle values at intervals of 45 degrees.

[0148] Of course, in another implementation, after the electronic device obtains the orientation category based on the largest confidence value, the ratio r between the second largest confidence value and the largest confidence value can be used to correct the final human body orientation angle to improve the accuracy of the orientation angle determination. For example, when r ≥ 0.95, the human body orientation angle is corrected by 22.5 degrees in the direction corresponding to the second largest confidence value; when r ≤ 0.5, the correction amount decays to 0; when 0.5 < r < 0.95, according to the position of r in the interval (0.5, 0.95), the correction amount is linearly mapped between 0 and 22.5 degrees. In this way, through the orientation angle correction, the electronic device can improve the accuracy of the orientation angle determination.

[0149] In this step, after the electronic device determines the first orientation angle of the first user and the second orientation angle of the second user, it can further determine whether the first orientation angle and the second orientation angle meet the preset angle conditions. If they do, the electronic device can determine that the image frame includes a reception behavior and belongs to a reception image frame. Exemplarily, in an image frame of the target image pair, if the angle differences between the first orientation angle, the second orientation angle and the direction of the line connecting the two parties are all less than the first preset angle threshold (such as 20 degrees, etc.), at this time, the first user and the second user are in a "face-to-face" state, or the difference between the first orientation angle and the second orientation angle is less than the second preset angle threshold (such as 30 degrees, etc.), at this time, the first user and the second user are in a "side-by-side" state. The electronic device can determine that there is a reception behavior in this image frame, and this image frame is a reception image frame.

[0150] S204 , aggregate reception image frames of the same target user pair to obtain a first reception segment.

[0151] In an embodiment of the present application, for a target user pair, the electronic device may aggregate reception image frames whose interval time is less than a preset time threshold (e.g., 5 seconds), and aggregate single-frame reception image frames to form a continuous reception behavior segment, namely, the first reception segment.

[0152] In the embodiment of the present application, the electronic device first screens the target user pair by the distance of the moving trajectory, then identifies the reception image frame with reception behavior according to whether the orientation angle meets the preset angle condition, and finally obtains the first reception segment corresponding to the target user pair through aggregation processing. In this way, the electronic device can achieve comprehensive and accurate recognition of the first reception segment in the video stream through distance screening and angle judgment.

[0153] S205: Extract a first reception feature corresponding to the first reception segment.

[0154] In a possible implementation, the first reception feature in step S205 can be extracted through the following steps (1) to (5):

[0155] (1) Determine the start and end time of the first reception corresponding to the first reception segment.

[0156] In an embodiment of the present application, after the electronic device determines the first reception segment in the video stream, it can determine the start time of the reception behavior according to the start frame time of the first reception segment, and can determine the end time of the reception behavior according to the end frame of the first reception segment, so that the first reception start and end times corresponding to the first reception segment can be obtained.

[0157] (2) Determine the first action type and the first action time corresponding to the first reception segment according to a preset action detection model.

[0158] In the embodiment of the present application, the preset action detection model may refer to a pre-set video action detection model, which can detect the time when the action occurs, the position of the action and the type of action. The preset action detection model may be a continuous multi-frame action detection algorithm (You Only Watch Once, YOWO), etc. Based on the preset action detection model, the electronic device can determine the first action type and the first action time corresponding to the first reception clip. It should be noted that in different offline retail scenarios, the first action type may be different. For example, in a car sales store, the first action type may include opening the car door, entering the car, opening the trunk, opening the hood, sitting down and talking, etc.; in a clothing store, the first action type may include picking up the goods, entering the dressing room, leaving the dressing room, trying on in front of the fitting mirror, etc.; in addition, the first action type may also include other action types in other scenes, which are not limited in the embodiment of the present application.

[0159] (3) Determine the first customer attribute corresponding to the first reception segment based on the human features of the second user in the first reception segment.

[0160] In the embodiment of the present application, the human body features may refer to the human body appearance features of the second user in the first reception segment, specifically, they may refer to head features, face features, etc. The embodiment of the present application does not limit the specific types of human body features. Based on the human body features of the second user in the first reception segment, the electronic device can identify the gender, age group and other attributes of the second user through a gender recognition model and an age recognition model, and obtain the first customer attributes corresponding to the first reception segment. Of course, the first customer attributes may also include other types of attributes, which can be identified through other models, and the embodiment of the present application does not limit this.

[0161] (4) Determine the first object attribute and the first object interaction time corresponding to the first reception segment according to the preset object detection model.

[0162] In the embodiment of the present application, the preset item detection model may refer to a model such as a pre-trained item detector or classifier. In an offline retail scenario, the reception behavior of the first user may be associated with a specific item. Taking a car sales store as an example, the reception behavior in the first reception segment usually occurs near the vehicle. When the coordinate positions of the first user and the second user in the first reception segment are near the vehicle, the electronic device can identify the color and model of the vehicle, and obtain the first item attribute corresponding to the first reception segment, which may include the item type of the vehicle and the attributes such as the color and model of the vehicle; at the same time, the electronic device can determine the moment when the distance between the first user and the vehicle is less than the preset item distance threshold as the first item interaction moment corresponding to the first item attribute, and the preset item distance threshold can be 5 times the width of the first user's human body detection frame, and of course it can also be other values. Of course, the electronic device can also use other methods to determine the first item attribute and the first item interaction moment, which is not limited in the embodiment of the present application.

[0163] (5) Determine the first clerk feature corresponding to the first reception clip based on the preset human body detection model.

[0164] In the embodiment of the present application, the electronic device can extract the appearance features of the first user, such as facial features, head features, etc., from the first reception clip to obtain the first clerk features corresponding to the first reception clip.

[0165] It should be noted that the above steps (1) to (5) are in an and / or relationship, that is, the electronic device can obtain at least one of the first reception start and end time, the first action type and the first action time, the first customer attribute, the first item attribute and the first item interaction time, and the first clerk feature as the first reception feature. Of course, the electronic device can also obtain other features as the first reception feature, and the embodiment of the present application is not limited to this.

[0166] Based on the above embodiments, Figure 3 A flowchart of determining the second reception segment and extracting the second reception feature provided by an exemplary embodiment of the present application. Figure 3 The second reception segment determination and second reception feature extraction process may specifically include:

[0167] S301, performing text conversion processing on an audio stream to obtain text sequence data corresponding to the audio stream.

[0168] In the embodiment of the present application, the text sequence data may refer to the time-sequential text data corresponding to the audio stream. Specifically, after the electronic device obtains the audio stream sent by the audio acquisition device, it can convert the audio stream into text through speech recognition technology, convert the speech signal into text information, and obtain the text sequence data corresponding to the audio stream.

[0169] In addition, the electronic device can further determine the speaker of each sentence, that is, the source of each voice signal, based on the audio acquisition device, such as the microphone array on the smart work badge, through the sound source localization technology, whether it is the first user wearing the audio acquisition device, or the person opposite the audio acquisition device (which may be the second user, or other first users, etc.), to achieve accurate association between audio and user, and ensure accurate identification of the subsequent second reception segment and the second reception features. It should be noted that speech recognition technology and sound source localization technology can use a variety of types of algorithms or tools, which can be flexibly selected based on actual needs, and this is not limited in the embodiments of the present application.

[0170] S302: Input the text sequence data into a preset text recognition model to determine the reception behavior corresponding to the text sequence data and the start and end times of the reception behavior.

[0171] In the embodiment of the present application, the preset text recognition model may refer to a pre-set model for recognizing text, which can determine the reception behavior in the text sequence data and the start and end time of the reception behavior. The preset text recognition model may be a text classification model or a large language dialogue model, etc.

[0172] Specifically, in offline retail scenarios, the beginning and the number of sections of the first user's reception behavior usually contain relatively fixed text content, such as starting with "Hello" and ending with "Goodbye". In a car sales store, when the first user begins to receive the second user, greetings such as "Which car are you interested in" or "Welcome" are usually involved; at the end of the reception, it is usually mentioned that "Contact me when you need it" and "Look forward to your visit again" and other content. In addition, when no conversation occurs for a long time, it can also be determined that the reception behavior has ended. On this basis, the electronic device can extract features based on the pre-trained model (Bidirectional Encoder Representations from Transformers, BERT) to train a special classifier to obtain a preset text recognition model, and use the preset text recognition model to identify the reception behavior in the second reception segment and determine the start and end time of the reception behavior; or, the electronic device can use the large prophecy dialogue model as a preset text recognition model, and identify the reception behavior in the second reception segment and determine the start and end time of the reception behavior by inputting the target prompt word. For example, the target prompt word can include the following content:

[0173] "You are a conversation analyst. Please read carefully the following conversation, which takes place in a car dealership and is approximately one minute long.

[0174] Each line of conversation content begins with a timestamp in the format of HH:MM:SS. The following symbol [] gives the identity of the speaker, which can be either a clerk or a person who can be a customer or other staff member of a car dealership. The following symbol '' contains the content of the conversation.

[0175] If the content in the symbol '' is VOID, it means that there is no conversation content within 10 seconds before the timestamp of this line.

[0176] Please read each line of dialogue carefully and judge whether this dialogue takes place between a clerk and a customer based on common sense of the sales process in a car sales store.

[0177] Then we can further determine whether the first sentence in the conversation (or the last non-VOID sentence) occurs when the clerk welcomes the customer and begins to serve (or when the clerk ends the service process).

[0178] The conversation begins:

[0179] <timestamp>[<speaker>]'<sentence>'

[0180] <timestamp>[<speaker>]'<sentence>'

[0181] <timestamp>[<speaker>]'<sentence>'

[0182] <timestamp>[<speaker>]'<sentence>'

[0183] <Timestamp>

&lt;Speaker&gt;

[0184] <Timestamp>

&lt;Speaker&gt;

[0186] End of conversation."

[0187] For reference, in the car dealership where this conversation took place, according to the service script requirements, the standard welcome phrases are <Standard Phrases 1>, and the standard farewell phrases are <Standard Phrases 2>. "

[0188] In the above target prompt words, the input conversation content is intercepted from the text sequence data corresponding to the second user using a sliding window of 1 minute in length, where the angle bracket fields are placeholders and can be replaced by actual content. By inputting the target prompt words into a preset text recognition model, the electronic device can identify each reception behavior included in the text sequence data and the start and end times of the reception behavior corresponding to the reception behavior. It should be noted that the electronic device can also use other models as preset text recognition models, or can also use other target prompt words, etc., which can be selected based on actual needs, and the embodiments of the present application are not limited to this.

[0189] S303: Determine, according to the start and end time of the reception behavior, the second reception segment corresponding to each first user in the audio stream.

[0190] In the embodiment of the present application, after determining the start and end time of the reception behavior, the electronic device can determine the second reception segment corresponding to each first user in the audio stream according to the start and end time of the reception behavior, thereby achieving comprehensive and accurate identification of the second reception segment.

[0191] S304: Extract a second reception feature corresponding to the second reception segment.

[0192] In a possible implementation, the second reception feature in step S304 can be implemented through the following steps (6) to (9):

[0193] (6) Determine the start and end time of the second reception corresponding to the second reception segment.

[0194] In the embodiment of the present application, the electronic device may use the start and end times of the reception behavior determined by the preset text recognition model as the second reception start and end times corresponding to the second reception segment.

[0195] (7) Determine the second action type and the second action time corresponding to the second reception segment according to the preset text recognition model.

[0196] In an embodiment of the present application, in an offline retail scenario, the reception process of the staff is often accompanied by some interactive actions. When these interactive actions occur, they may be mentioned in the conversation between the staff and the customer. The electronic device can identify the second action type and the second action time in the second reception segment based on a preset text recognition model. Taking the preset text recognition model as a large prediction dialogue model as an example, the electronic device can input an action recognition prompt word in the preset text recognition model, and the preset text recognition model can output the second action type and the second action time corresponding to the second reception segment. Exemplarily, the action recognition prompt word may include the following content:

[0197] "You are a conversation analyst. Please read carefully the following conversation, which takes place in a car dealership and is approximately one minute long.

[0198] Each line of conversation begins with a timestamp in the format of HH:MM:SS. In the following [], the speaker's identity is given. There are two possible values: clerk or customer. In the following '', the content of the conversation is given.

[0199] After reading each line of dialogue carefully, please judge based on the common sense of the sales process in a car sales store whether the customer or the clerk made an <action> in this dialogue; if so, determine the time when the <action> occurred.

[0200] The conversation begins:

[0201] <timestamp>[<speaker>]'<sentence>'

[0202] <timestamp>[<speaker>]'<sentence>'

[0203] <timestamp>[<speaker>]'<sentence>'

[0204] <timestamp>[<speaker>]'<sentence>' ...

[0206] End of conversation."

[0207] In the action recognition prompt, the <action> placeholder can be replaced by action types such as opening the door, entering the car, opening the trunk, opening the hood, sitting down to talk, etc. In this way, the electronic device can accurately recognize the second action type and the second action time in the second reception segment by inputting the action recognition prompt and analyzing and processing it through the preset text recognition model.

[0208] (8) Determine the second customer attribute corresponding to the second reception segment based on the original audio signal corresponding to the second reception segment and the preset sound recognition model.

[0209] In an embodiment of the present application, the original audio signal may refer to an audio signal in an audio stream corresponding to the second reception segment. The preset sound recognition model may refer to a preset sound classifier model that can identify the second user's gender, age group and other attributes. The electronic device can analyze and identify the original audio signal corresponding to the second reception segment through a preset sound recognition model. Specifically, the Mel-Frequency Cepstral Coefficients (MFCC) features can be extracted from the original audio signal, and then a support vector machine (SVM) classifier can be used to classify the gender and age group to obtain the second customer attribute corresponding to the second user in the second reception segment.

[0210] (9) Determine the second object attribute and the second object interaction time corresponding to the second reception segment according to the preset text recognition model.

[0211] In an embodiment of the present application, in an offline retail scenario, reception behavior may be associated with specific items. For example, in a car sales store, customers and staff will talk about the display car they are watching. The model and color of the vehicle are the second item attributes corresponding to the second reception segment, and the moment when the model and color are discussed is the second item interaction moment. Based on the text sequence data of the second reception segment, the electronic device can determine the second item attributes and the second item interaction moment included in the second reception segment based on a preset text recognition model. Taking the preset text recognition model as the large prophecy dialogue model as an example, the electronic device can input item recognition prompt words in the preset text recognition model, and the preset text recognition model can output the second item attributes and the second item interaction moment corresponding to the second reception segment. Exemplarily, the item recognition prompt words may include the following content:

[0212] "You are a conversation analyst. Please read carefully the following conversation, which takes place in a car dealership and is approximately one minute long.

[0213] Each line of conversation begins with a timestamp in the format of HH:MM:SS. The following [] gives the identity of the speaker, which can be either a clerk or a customer. The following '' contains the content of the conversation.

[0214] After reading each line of the conversation carefully, please judge based on the common sense of the sales process in a car sales store whether the customer or the clerk is talking about the <attribute> item in this conversation; if so, determine the time when the <attribute> item is being discussed.

[0215] Please note that you should only answer affirmatively if you are certain that the customer is standing next to the model of the display car they are talking about. Do not answer affirmatively if the car the customer and the salesperson are talking about may not be next to them.

[0216] The conversation begins:

[0217] <timestamp>[<speaker>]'<sentence>'

[0218] <timestamp>[<speaker>]'<sentence>'

[0219] <timestamp>[<speaker>]'<sentence>'

[0220] <timestamp>[<speaker>]'<sentence>' ...

[0222] End of conversation."

[0223] In the above-mentioned item recognition prompt, the <attribute> placeholder can be replaced by the specific attribute of the item. In this way, the electronic device can accurately recognize the second item attribute and the second item interaction moment in the second reception segment by inputting the item recognition prompt and analyzing and processing it through the preset text recognition model.

[0224] It should be noted that the above steps (6) to (9) are in an and / or relationship, that is, the electronic device can obtain at least one of the second reception start and end time, the second action type and the second action time, the second customer attribute, the second item attribute and the second item interaction time as the second reception feature. Of course, the electronic device can also obtain other features as the second reception feature. The embodiment of the present application does not limit this. However, the first reception feature and the second reception feature obtained by the electronic device in the embodiment of the present application need to include features of the same type to ensure the rationality of the subsequent target relevance calculation.

[0225] Based on the above embodiments, Figure 4 A flowchart of a target reception segment and a target user identification determination process provided by an exemplary embodiment of the present application. Figure 4 The second reception segment determination and second reception feature extraction process may specifically include:

[0226] S401: Determine the start and end time correlation between the first reception start and end time and the second reception start and end time.

[0227] In the embodiments of the present application, the relevance of start and end times may refer to the relevance between the start and end times of the first reception segment and the start and end times of the second reception segment, which can be represented by score1. This relevance of start and end times may consist of two parts: the relevance of the reception start time score1_1 and the relevance of the reception end time score1_2. For example, this relevance of start and end times may be a weighted sum of the relevance of the reception start time and the relevance of the reception end time, that is, score1 = b1 * score1_1 + b2 * score1_2. Among them, b1 and b2 are weight coefficients, which may be preset values. For example, b1 is 0.8 and b2 is 0.2, etc.

[0228] Specifically, the relevance of the reception start time score1_1 can be calculated in the following way: when |start1_1 - start1_2| < thr_1, score1_1 = exp(-k1 * |start1_1 - start1_2|); otherwise, score1_1 = 0. Among them, |start1_1 - start1_2| is the absolute value of the difference between the start time of the reception behavior in the first reception start and end times and the start time of the reception behavior in the second reception start and end times; thr_1 is a preset difference threshold, which may be 60 seconds, etc.; exp is an exponential function; k1 is the first preset parameter of the exponential decay function, which may be 0.05, etc. Similarly, the relevance of the reception end time score1_2 can be calculated in a similar way, and the embodiments of the present application will not elaborate on this.

[0229] It should be noted that for the first reception segment and the second reception segment, the electronic device may perform the calculations of the subsequent action relevance, customer attribute relevance, item relevance, and store clerk relevance only when the relevance of the start and end times is greater than 0, and do not perform the calculations of the subsequent action relevance, customer attribute relevance, item relevance, and store clerk relevance when the relevance of the start and end times is equal to 0, and directly determine that the target relevance is 0. This can avoid wasting computing resources and improve the computing efficiency of the target relevance.

[0230] S402. Determine the action relevance according to the matching result of the first action type and the second action type, the first action time, and the second action time.

[0231] In the embodiment of the present application, the action correlation may refer to the correlation between the first action type and the first action moment in the first reception feature and the second action type and the second action moment in the second reception feature, and the action correlation may be represented by score2. Specifically, if the first action type of the first reception segment is different from the second action type of the second reception segment, or if at least one of the first reception feature and the second reception feature does not recognize the action type, the action correlation score2 is equal to 0. If the first action type of the first reception segment is the same as the second action type of the second reception segment, the electronic device may calculate the action correlation in the following manner: score2=exp(-k2*|t2_1-t2_2|). Among them, |t2_1-t2_2| is the absolute value of the difference between the first action moment and the second action moment, and k2 is the second preset parameter of the exponential decay function, which may be 0.05 or the like.

[0232] S403: Determine the customer attribute relevance according to the matching results of the first customer attribute and the second customer attribute.

[0233] In the embodiment of the present application, the customer attribute relevance may refer to the degree of relevance between the first customer attribute of the first reception segment and the second customer attribute of the second reception segment, and the customer attribute relevance may be represented by score 3. If the gender of the second user in the first customer attribute is the same as the gender of the second user in the second customer attribute, the electronic device may determine that the customer attribute relevance score 3 is equal to 1; if the gender of the second user in the first customer attribute is the same as the gender of the second user in the second customer attribute, the electronic device may determine that the customer attribute relevance score 3 is equal to.

[0234] On the basis of the above-mentioned gender matching, if the age group of the second user in the first customer attribute and the age group of the second user in the second customer attribute are both higher than the preset confidence threshold, and the two age groups are different, the electronic device can subtract a preset value, such as 0.5, from the customer attribute relevance score3; if at least one of the age group of the second user in the first customer attribute and the age group of the second user in the second customer attribute is not higher than the preset confidence threshold, or the two age groups are the same, the electronic device can maintain the customer attribute relevance score3 unchanged.

[0235] It should be noted that, since the first customer attribute and the second customer attribute may also include other attributes, the electronic device may calculate the customer attribute relevance in other ways, such as weighted summation, etc., which is not limited in the embodiments of the present application.

[0236] S404: Determine the item relevance according to the matching result of the first item attribute and the second item attribute, the first item interaction time, and the second item interaction time.

[0237] In the embodiment of the present application, the item relevance may be the relevance of the items between the first reception segment and the second reception segment, and the item relevance may be represented by score 4. Similar to the action relevance, if the first item attribute is different from the second item attribute, or at least one of the first reception feature and the second reception feature does not include the item attribute, the electronic device may determine that the items in the first reception segment and the second reception segment are different, and the item relevance score 4 is equal to 0.

[0238] If the first item attribute is the same as the second item attribute, the electronic device may determine the item relevance in the following manner: score4=exp(-k4*|t4_1-t4_2|). Wherein, |t4_1-t4_2| is the absolute value of the difference between the first item interaction time and the second item interaction time, and k4 is the fourth preset parameter of the exponential decay function, which may be 0.02 or the like.

[0239] S405: Determine the store clerk relevance according to the similarity between the first store clerk feature and the preset store clerk features in the preset database.

[0240] In the embodiment of the present application, the clerk relevance may refer to the correlation between the appearance features of the first user in the first reception segment of the video stream and the appearance features of the clerk in the preset database, and the clerk relevance may be represented by score 5. The preset database refers to a preset clerk data storage database, which includes preset clerk features, and the preset clerk features may refer to the actual appearance features of each first user entered in advance, such as facial features, head features, etc.

[0241] Specifically, after determining the first clerk feature in the first reception segment of the video stream, the electronic device can determine the preset clerk feature corresponding to the first user identifier according to the first user identifier corresponding to the second reception segment, and then the electronic device can calculate the similarity between the first clerk feature and the preset clerk feature. Specifically, the vector similarity between the first clerk feature and the preset clerk feature can be calculated. If the similarity is greater than a preset similarity threshold (for example, 0.6), the electronic device can determine that the clerk relevance score5 is equal to 1; if the similarity is less than or equal to the preset similarity threshold, the electronic device can determine that the clerk relevance score5 is equal.

[0242] S406. Determine a target correlation between the first reception feature and the second reception feature according to at least one of a start and end time correlation, an action correlation, a customer attribute correlation, an item correlation, and a clerk correlation.

[0243] In the embodiment of the present application, the electronic device can calculate the target correlation between the first reception feature and the second reception feature of any first reception segment and the second reception feature of the second reception segment, and obtain the target correlation between the first reception segment and the second reception segment. Specifically, after calculating the start and end time correlation, action correlation, customer attribute correlation, item correlation and clerk correlation of the first reception feature and the second reception feature, the electronic device can obtain the target correlation between the first reception feature and the second reception feature by performing a comprehensive calculation on these correlations. The electronic device can specifically calculate the target correlation between the first reception feature and the second reception feature by weighted summation, principal component analysis, machine learning and other methods, and can also calculate the target correlation between the first reception feature and the second reception feature by fusion methods such as neural networks or data models such as classification regression trees, and the embodiment of the present application does not limit this.

[0244] Exemplarily, the electronic device may calculate the target relevance by weighted summation, and the target relevance score may be calculated as follows:

[0245] score=a1*score1+a2*score2+a3*score3+a4*score4+a5*score5

[0246] Among them, a1, a2, a3, a4, and a5 may refer to the weights corresponding to the correlation of each type, and the specific values ​​may be set based on actual needs, for example, the values ​​may be 0.4, 0.1, 0.1, 0.1, and 0.3, respectively. Of course, other values ​​may also be taken, and the embodiments of the present application do not limit this. In this way, the electronic device can calculate the target correlation between the first reception feature and the second reception feature by comprehensively calculating the correlation of the start and end times, the correlation of actions, the correlation of customer attributes, the correlation of items, and the correlation of store clerks, and can achieve accurate reflection of the first reception segment and the second reception segment, thereby improving the accuracy of subsequent reception segment matching.

[0247] For example, Figure 5 The following is a schematic diagram of a target relevance calculation process provided by an exemplary embodiment of the present application. Figure 5 As shown, the electronic device can extract the first reception start and end time, the first action type and the first action time, the first customer attribute information, the first item attribute and the first item interaction time, and the first clerk feature from the first reception segment of the video stream to obtain the first reception feature corresponding to the first reception segment. The electronic device can also extract the second reception start and end time, the second action type and the second action time, the second customer attribute information, the second item attribute and the second item interaction time from the second reception segment of the audio stream to obtain the second reception feature corresponding to the second reception segment.

[0248] Afterwards, the electronic device can calculate the start and end time correlation, action correlation, customer attribute correlation, and item correlation between the first reception feature and the second reception feature, and can also determine the clerk correlation. The electronic device can perform comprehensive calculations on different types of correlations, and ultimately obtain the target correlation between the first reception segment and the second reception segment.

[0249] It should be emphasized that when the electronic device extracts the first reception feature from the first reception segment and extracts the second reception feature from the second reception segment, it can extract Figure 5 Correspondingly, when calculating the target relevance, the electronic device can perform calculations based on the relevance corresponding to the partial features, for example, the target relevance can be calculated based on the start and end time relevance and the action relevance, etc., which can be selected based on the actual scenario, and the embodiment of the present application is not limited to this.

[0250] S407: Match the first reception segment and the second reception segment according to the target relevance, and determine the target reception segment and the target user identifier corresponding to each target reception behavior.

[0251] In an embodiment of the present application, after determining the target similarity between the first reception segment and the second reception segment, the electronic device can match the first reception segment and the second reception segment based on the target similarity to determine the target reception segment and the target user identifier corresponding to the same target reception behavior.

[0252] For example, Figure 6 A timing diagram of a first reception segment and a second reception segment provided by an exemplary embodiment of the present application. Figure 6 As shown, the horizontal direction is the time axis, and each solid rectangle indicates the time range in which a second user (customer) appears in the store. Taking six second users who enter and leave the store successively as an example, they are respectively recorded as G1, G2, G3, G4, G5, and G6. The time range can be determined according to the second movement trajectory of the second user in the video stream. Figure 6 Each dotted rectangle represents the time range in which a first user (staff member) appears in the store. Taking three first users as an example, they are respectively recorded as H1, H2, and H3. The first user identifier of each first user can be determined by the first user identifier associated with the device identifier of the audio acquisition device.

[0253] exist Figure 6In , the shaded portion with a serial number in each rectangular box can represent the reception segment determined by the electronic device. For example, the shaded portion in the solid line rectangular box is the first reception segment in the video stream, and the shaded portion in the dashed line rectangle is the second reception segment in the audio stream. After the electronic device extracts the first reception feature of the first reception segment and the second reception feature of the second reception segment, it can calculate the target correlation between the first reception feature and the second reception feature, and then associate and match the first reception segment and the second reception segment according to the target correlation to determine the target reception segment corresponding to each target reception behavior, and at the same time determine whether the first user corresponding to the target reception behavior is H1, H2, or H3. Figure 6 , the final association and matching result of the electronic device, that is, the target reception segment corresponding to the same target reception behavior, can be: G1_1 is associated with H3_1, G2_1 is associated with H1_2 and H1_3, G3_1 is associated with H1_1, G4_1 is associated with H2_1, G6_1 and G6_2 are associated with H3_2, and G6_3 is associated with H2_3. A total of 6 target reception behaviors occurred (if the user identifiers of the first user and the second user in two consecutive target reception behaviors are the same, it is recorded as 1 target reception behavior).

[0254] also, Figure 1 G4_2 in the figure indicates the misdetected reception time period of the second user G4, i.e., the misdetected first reception segment. Of course, it is also possible that the second reception segment is missed when the audio stream detects it, resulting in the absence of the second reception segment corresponding to the first reception segment G4_2. Since there is no second reception segment that can be matched, the first reception segment G4_2 can be filtered out during the association matching process. Similarly, Figure 1 The H2_2 in the figure represents the false detection time period of the reception behavior of the first user H2, that is, the falsely detected second reception segment. Of course, it may also be that there is a missed detection when the video stream detects the first reception segment, resulting in the non-existence of the first reception segment corresponding to the second reception segment H2_2. Since there is no matching first reception segment, the second reception segment H2_2 can be filtered out during the association matching process.

[0255] In the embodiment of the present application, when the electronic device performs association matching on the first reception segment and the second reception segment based on the target association, a variety of matching methods may be used, for example, association matching may be performed directly based on the size of the target correlation, or association matching may be performed based on an optimization analysis model. In a possible implementation, step S407 may be specifically implemented through the following steps (10) to (11):

[0256] (10) According to the target relevance between the first reception feature and the second reception feature, the first reception segment and the second reception segment having the greatest target relevance to the first reception segment are determined as target reception segments corresponding to the target reception behavior.

[0257] (11) Determine the first user identifier corresponding to the second reception segment in the target reception segment as the target user identifier.

[0258] In the embodiment of the present application, after calculating the target correlation between the first reception segment and the second reception segment, the electronic device can directly determine the first reception segment and the second reception segment with the largest target correlation with the first reception segment as target reception segments corresponding to the same target reception behavior according to the size of the target correlation, and then can determine the target user identifier corresponding to the target reception behavior according to the first user identifier corresponding to the second reception segment, thereby determining the identity of the staff in the target reception behavior. In this way, the electronic device can achieve rapid matching of the first reception segment and the second reception segment according to the size of the target correlation, thereby improving the recognition efficiency of the reception behavior.

[0259] In another possible implementation, step S407 may be implemented by the following steps (12) to (13):

[0260] (12) Determine a correlation matrix corresponding to the first reception segment and the second reception segment according to the target correlation between the first reception feature and the second reception feature; the elements in the correlation matrix are the target correlations.

[0261] (13) Delete the elements whose target relevance is less than a preset relevance threshold from the relevance matrix to obtain an updated relevance matrix; match the first reception segment and the second reception segment according to the preset planning model for the updated relevance matrix to determine the target reception segment and the target user identifier corresponding to each target reception behavior.

[0262] In the embodiment of the present application, the correlation matrix may refer to a matrix composed of target correlations between the first reception segment and the second reception segment. The preset correlation threshold may refer to a preset correlation critical value, such as 0.2. The preset planning model may refer to a preset optimization technology model, which may specifically be a linear programming model, etc. The embodiment of the present application does not limit the specific type of the preset planning model.

[0263] In the process of associating and matching the first reception segment and the second reception segment according to the target relevance, the method of directly matching according to the maximum value of the target relevance may have certain matching errors. In this step, the electronic device can generate a relevance matrix according to the target relevance, and then update the relevance matrix according to the preset relevance threshold. Specifically, the elements in the relevance matrix that are less than the preset relevance threshold can be set to 0 to obtain an updated relevance matrix. Subsequently, the electronic device can globally optimize the updated relevance matrix based on the preset planning model to obtain the matching result of the first reception segment and the second reception segment, determine the target reception segment and the target user identifier corresponding to each target reception behavior, and further improve the rationality and accuracy of reception behavior recognition.

[0264] In a possible implementation, step (13) can be implemented by the following steps (14) to (15):

[0265] (14) Determine the decision variables corresponding to each non-zero element in the updated correlation matrix; determine the objective function and the objective constraint conditions corresponding to the objective function based on each decision variable and the non-zero elements corresponding to the decision variable.

[0266] (15) Calculate the objective function according to the objective constraint conditions to obtain the target value corresponding to each decision variable; and determine the target reception segment and target user ID corresponding to each target reception behavior according to the target value.

[0267] In an embodiment of the present application, a decision variable may refer to an unknown variable corresponding to the updated correlation matrix, and the target value of the decision variable may be 1 or 0. The objective function may refer to a linear programming objective function determined based on a preset planning model and decision variables. The target constraint may refer to a constraint relationship between decision variables. Since the same first user cannot receive different second users at the same time, and the same second user cannot be received by different first users at the same time, the electronic device may determine the target constraint corresponding to the target function based on the time overlap relationship between the first reception segment and the second reception segment.

[0268] Specifically, after obtaining the updated correlation matrix, the electronic device can determine the decision variables in the updated correlation matrix, and determine the objective function based on the decision variables and the preset planning model, and at the same time determine the target constraint relationship corresponding to the objective function, and then solve and calculate the objective function based on the target constraint relationship to obtain the target value of each decision variable, and finally determine the associated matching first reception segment and second reception segment based on the target value, and then obtain the target reception segment and target user identification corresponding to the target reception behavior. In this way, the electronic device establishes the objective function based on the preset planning model, and solves it in combination with the target constraint conditions, and can achieve accurate matching of the first reception segment and the second reception segment based on linear programming, which can improve the accuracy of reception behavior recognition.

[0269] For example, Figure 7 A schematic diagram of the association and matching between a first reception segment and a second reception segment provided by an exemplary embodiment of the present application. Figure 7 As shown, Figure 6 Based on the first reception segments and the second reception segments in the electronic device, after determining the target correlation between the first reception feature and the second reception feature, the electronic device can obtain the correlation matrix (a) corresponding to the first reception segment and the second reception segment. Each element in the correlation matrix (a) is the target correlation between the first reception segment and the second reception segment corresponding to the row and column where the element is located. The electronic device can first delete the elements less than the preset correlation threshold in the correlation matrix (a) to obtain the updated correlation matrix (b).

[0270] Afterwards, the electronic device determines the decision variables corresponding to each non-zero element in the updated correlation matrix (b), recorded as x1 to x13, and obtains the decision variable matrix (c), where the value of the decision variable can be 0 or 1. The value of the decision variable is 1, indicating that a reception behavior has occurred between the first user and the second user corresponding to the row and column where the decision variable is located, and the value is 0, indicating that no reception behavior has occurred. Based on these decision variables and the preset planning model, the electronic device can use, for example, a 0-1 linear programming method to determine the objective function z as:

[0271] z=x1*0.75+x2*0.69+x3*0.54+x4*0.56+x5*0.71+x6*0.76+x7*0.33+x8*0.31+x9*0.92+x10*0.85+x11*0.24+x12*0.38+x13*0.45

[0272] Furthermore, since the same first user cannot receive different second users at the same time, and the same second user cannot be received by different first users at the same time, the electronic device can determine that the target constraint conditions corresponding to the target function include: x2+x5=1 (i.e., in the G2_1 time period, G2 will not be received by H1 and H2 at the same time, so only one of the two decision variables x2 and x5 can be 1), x4+x7=1, x1+x11=1, x3+x6=1, x2+x3=1, x5+x6=1, and x8+x9=1. It should be noted that offline retail scenarios are more complex, and electronic devices can also set other constraints, such as the presence of multiple staff members receiving at the same time, which can be set based on the actual scenario, and the embodiments of the present application are not limited to this.

[0273] Then, the electronic device calculates and solves the objective function according to the objective constraints, and the optimization result is as follows: Figure 7 As shown in the matrix (d) in , the target values ​​of each decision variable are finally obtained, and then the target reception segment corresponding to the same target reception behavior and the target user identifier corresponding to the target reception behavior can be determined. The electronic device can specifically use linear programming tool software to perform calculation and solution, and the embodiment of the present application does not limit the specific implementation method of the calculation and solution.

[0274] It should be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0275] Based on any one of the above embodiments, Figure 8 A processing logic diagram of reception behavior recognition provided by an exemplary embodiment of the present application. Figure 8 As shown, the electronic device can obtain the video stream sent by the shooting device and the audio stream sent by the audio acquisition device. For the video stream, the electronic device can identify the first reception segment with reception behavior in the video stream through video information, and extract the first reception feature corresponding to the first reception segment; for the audio stream, the electronic device can identify the second reception segment with reception behavior in the audio stream through voice recognition, and extract the second reception feature corresponding to the first reception segment. Afterwards, the electronic device can calculate the target correlation between the first reception feature and the second reception feature, and associate and match the first reception segment and the second reception segment according to the target correlation, and finally obtain the target reception segment corresponding to the same target reception behavior and the target user identifier corresponding to the target reception behavior.

[0276] In the embodiment of the present application, the electronic device correlates and matches the features of the two sources of video stream and audio stream, and adopts multiple types of features, which can improve the accuracy of reception behavior recognition and the accuracy of staff identity recognition in reception behavior; and using video information, it can identify the total number of customers in the venue, including customers who have not been received, so as to more completely reflect the reception situation of the staff; in addition, the electronic device does not need to adopt indoor positioning technology that increases the system cost, nor does it require manual registration, which can reduce hardware costs and labor costs, and realize automatic and accurate recognition of reception behavior.

[0277] Fig. 9 For a structural diagram of a reception behavior recognition device provided by an exemplary embodiment of the present application, see Fig. 9 The reception behavior recognition device 90 includes:

[0278] The acquisition module 91 is used to acquire the video stream sent by the shooting device and the audio stream sent by the audio acquisition device; the device identifier of the audio acquisition device is associated with the first user identifier;

[0279] A first determining module 92, configured to determine a first reception segment in a video stream and extract a first reception feature corresponding to the first reception segment;

[0280] A second determining module 93, configured to determine a second reception segment in the audio stream and extract a second reception feature corresponding to the second reception segment;

[0281] The matching module 94 is used to match the first reception segment and the second reception segment according to the target correlation between the first reception feature and the second reception feature, and determine the target reception segment and the target user identifier corresponding to each target reception behavior.

[0282] In a possible implementation manner, the first determining module 92 is specifically configured to:

[0283] Determine a first movement trajectory of a first user and a second movement trajectory of a second user in a video stream;

[0284] Determining a target user pair according to a distance between the first moving trajectory and the second moving trajectory;

[0285] For each image frame corresponding to the target user pair, if in the image frame, a first orientation angle of a first user in the target user pair and a second orientation angle of a second user meet a preset angle condition, then the image frame is determined to be a reception image frame including a reception behavior;

[0286] Aggregate the reception image frames of the same target user pair to obtain a first reception segment.

[0287] In a possible implementation manner, the first determining module 92 is specifically configured to:

[0288] Determine the first reception start and end time corresponding to the first reception segment; and / or,

[0289] Determine, according to a preset action detection model, a first action type and a first action time corresponding to the first reception segment; and / or,

[0290] determining a first customer attribute corresponding to the first reception segment according to the body features of the second user in the first reception segment; and / or,

[0291] Determine, according to a preset object detection model, a first object attribute and a first object interaction time corresponding to the first reception segment; and / or,

[0292] According to the preset human body detection model, a first clerk feature corresponding to the first reception segment is determined.

[0293] In a possible implementation manner, the second determining module 93 is specifically configured to:

[0294] Perform text conversion processing on the audio stream to obtain text sequence data corresponding to the audio stream;

[0295] Input the text sequence data into a preset text recognition model to determine the reception behavior corresponding to the text sequence data and the start and end times of the reception behavior;

[0296] According to the start and end time of the reception behavior, the second reception segment corresponding to each first user in the audio stream is determined.

[0297] In a possible implementation manner, the second determining module 93 is specifically configured to:

[0298] Determine the second reception start and end time corresponding to the second reception segment; and / or,

[0299] Determine, according to a preset text recognition model, a second action type and a second action time corresponding to the second reception segment; and / or,

[0300] determining a second customer attribute corresponding to the second reception segment according to an original audio signal corresponding to the second reception segment and a preset sound recognition model; and / or,

[0301] According to the preset text recognition model, a second object attribute and a second object interaction time corresponding to the second reception segment are determined.

[0302] In a possible implementation, the device 90 is further used for:

[0303] Determine the start and end time correlation between the first reception start and end time and the second reception start and end time; and / or,

[0304] determining the action relevance according to the matching result of the first action type and the second action type, the first action time and the second action time; and / or,

[0305] Determining the customer attribute relevance according to the matching result of the first customer attribute and the second customer attribute; and / or,

[0306] Determining the item relevance according to the matching result of the first item attribute and the second item attribute, the first item interaction time and the second item interaction time; and / or,

[0307] Determining the store clerk relevance according to the similarity between the first store clerk feature and the preset store clerk features in the preset database;

[0308] A target correlation between the first reception feature and the second reception feature is determined based on at least one of a start and end time correlation, an action correlation, a customer attribute correlation, an item correlation, and a clerk correlation.

[0309] In a possible implementation, the matching module 94 is specifically configured to:

[0310] According to the target correlation between the first reception feature and the second reception feature, the first reception segment and the second reception segment having the greatest target correlation with the first reception segment are determined as target reception segments corresponding to the target reception behavior;

[0311] The first user identifier corresponding to the second reception segment in the target reception segment is determined as the target user identifier.

[0312] In a possible implementation, the matching module 94 is specifically configured to:

[0313] Determine a correlation matrix corresponding to the first reception segment and the second reception segment according to the target correlation between the first reception feature and the second reception feature; the elements in the correlation matrix are the target correlations;

[0314] Deleting elements whose target relevance is less than a preset relevance threshold from the relevance matrix to obtain an updated relevance matrix;

[0315] For the updated correlation matrix, the first reception segment and the second reception segment are matched according to the preset planning model to determine the target reception segment and the target user identifier corresponding to each target reception behavior.

[0316] In a possible implementation, the matching module 94 is specifically configured to:

[0317] Determine the decision variables corresponding to each non-zero element in the updated correlation matrix;

[0318] According to each decision variable and the non-zero elements corresponding to the decision variable, determine the objective function and the objective constraint conditions corresponding to the objective function;

[0319] Calculate the objective function according to the objective constraints to obtain the target values ​​corresponding to each decision variable;

[0320] According to the target value, the target reception segment and the target user identifier corresponding to each target reception behavior are determined.

[0321] The reception behavior recognition device 90 provided in the embodiment of the present application can execute the technical solution shown in the above method embodiment, and its implementation principle and beneficial effects are similar, which will not be repeated here.

[0322] Fig.10 For a schematic diagram of the structure of an electronic device provided by an exemplary embodiment of the present application, see Fig.10 The electronic device 100 may include a processor 1001 and a memory 1002. Exemplarily, the processor 1001 and the memory 1002 are interconnected via a bus 1003.

[0323] Memory 1002 stores computer executable instructions;

[0324] The processor 1001 executes the computer-executable instructions stored in the memory 1002 , so that the processor 1001 executes the reception behavior recognition method as shown in the above method embodiment.

[0325] Accordingly, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the reception behavior recognition method of the above method embodiment.

[0326] Accordingly, an embodiment of the present application may also provide a computer program product, including a computer program. When the computer program is executed by a processor, the reception behavior recognition method shown in the above method embodiment can be implemented.

[0327] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0328] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0329] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0330] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0331] In a typical configuration, a computing device includes one or more processors, input / output interfaces, network interfaces, and memory.

[0332] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0333] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined in this article, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0334] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0335] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A reception behavior recognition method, characterized in that: include: Get the video stream sent by the shooting device and the audio stream sent by the audio acquisition device; The device identification of the audio acquisition device is associated with the first user identification; Determining a first reception segment in the video stream, and extracting a first reception feature corresponding to the first reception segment; determining a second reception segment in the audio stream, and extracting a second reception feature corresponding to the second reception segment; According to the target correlation between the first reception feature and the second reception feature, the first reception segment and the second reception segment are matched to determine the target reception segment and the target user identifier corresponding to each target reception behavior.

2. The method according to claim 1, characterized in that The determining of the first reception segment in the video stream comprises: Determine a first movement trajectory of the first user and a second movement trajectory of the second user in the video stream; determining a target user pair according to a distance between the first moving trajectory and the second moving trajectory; For each image frame corresponding to the target user pair, if in the image frame, a first orientation angle of a first user and a second orientation angle of a second user in the target user pair meet a preset angle condition, then determining the image frame as a reception image frame including a reception behavior; Aggregate the reception image frames of the same target user pair to obtain the first reception segment.

3. The method according to claim 1, characterized in that The extracting the first reception feature corresponding to the first reception segment includes: Determine the first reception start and end time corresponding to the first reception segment; and / or, Determine, according to a preset action detection model, a first action type and a first action time corresponding to the first reception segment; and / or, determining a first customer attribute corresponding to the first reception segment according to the body features of the second user in the first reception segment; and / or, determining, according to a preset object detection model, a first object attribute and a first object interaction time corresponding to the first reception segment; and / or, According to a preset human body detection model, a first clerk feature corresponding to the first reception segment is determined.

4. The method according to claim 1, characterized in that: The determining of the second reception segment in the audio stream comprises: Performing text conversion processing on the audio stream to obtain text sequence data corresponding to the audio stream; Inputting the text sequence data into a preset text recognition model to determine the reception behavior corresponding to the text sequence data and the start and end times of the reception behavior; According to the start and end times of the reception behavior, a second reception segment corresponding to each first user in the audio stream is determined.

5. The method according to claim 1, characterized in that: The extracting the second reception feature corresponding to the second reception segment includes: Determine the second reception start and end time corresponding to the second reception segment; and / or, determining, according to a preset text recognition model, a second action type and a second action time corresponding to the second reception segment; and / or, determining a second customer attribute corresponding to the second reception segment according to an original audio signal corresponding to the second reception segment and a preset sound recognition model; and / or, According to the preset text recognition model, a second object attribute and a second object interaction time corresponding to the second reception segment are determined.

6. The method according to claim 1, characterized in that The method further comprises: Determine the start and end time correlation between the first reception start and end time and the second reception start and end time; and / or, determining the action relevance according to the matching result of the first action type and the second action type, the first action time and the second action time; and / or, Determining the customer attribute relevance according to the matching result of the first customer attribute and the second customer attribute; and / or, Determining the item relevance according to the matching result of the first item attribute and the second item attribute, the first item interaction time and the second item interaction time; and / or, Determining the store clerk relevance according to the similarity between the first store clerk feature and the preset store clerk features in the preset database; A target correlation between the first reception feature and the second reception feature is determined based on at least one of the start and end time correlation, the action correlation, the customer attribute correlation, the item correlation and the clerk correlation.

7. The method according to claim 1, characterized in that The matching of the first reception segment and the second reception segment to determine the target reception segment and the target user identifier corresponding to each target reception behavior includes: According to the target correlation between the first reception feature and the second reception feature, determining the first reception segment and the second reception segment having the greatest target correlation with the first reception segment as the target reception segment corresponding to the target reception behavior; The first user identifier corresponding to the second reception segment in the target reception segment is determined as the target user identifier.

8. The method according to claim 1, characterized in that The matching of the first reception segment and the second reception segment to determine the target reception segment and the target user identifier corresponding to each target reception behavior includes: Determine a correlation matrix corresponding to the first reception segment and the second reception segment according to the target correlation between the first reception feature and the second reception feature; the elements in the correlation matrix are the target correlations; Deleting the elements whose target relevance is less than a preset relevance threshold from the relevance matrix to obtain an updated relevance matrix; With respect to the updated correlation matrix, the first reception segment and the second reception segment are matched according to a preset planning model to determine the target reception segment and the target user identifier corresponding to each target reception behavior.

9. The method according to claim 8, characterized in that The matching of the first reception segment and the second reception segment according to the preset planning model to determine the target reception segment and the target user identifier corresponding to each target reception behavior includes: Determine the decision variables corresponding to each non-zero element in the updated correlation matrix; Determine an objective function and an objective constraint condition corresponding to the objective function according to each of the decision variables and the non-zero elements corresponding to the decision variables; Calculating the objective function according to the objective constraint condition to obtain the objective value corresponding to each of the decision variables; According to the target value, the target reception segment and the target user identifier corresponding to each target reception behavior are determined.

10. A reception behavior recognition device, characterized in that: include: An acquisition module is used to acquire the video stream sent by the shooting device and the audio stream sent by the audio acquisition device; The device identification of the audio acquisition device is associated with the first user identification; A first determining module, configured to determine a first reception segment in the video stream, and extract a first reception feature corresponding to the first reception segment; A second determining module, configured to determine a second reception segment in the audio stream, and extract a second reception feature corresponding to the second reception segment; The matching module is used to match the first reception segment and the second reception segment according to the target correlation between the first reception feature and the second reception feature, and determine the target reception segment and the target user identifier corresponding to each target reception behavior.

11. An electronic device, characterized in that: include: Memory and processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor executes the reception behavior recognition method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the reception behavior recognition method according to any one of claims 1 to 9 when executed by a processor.

13. A computer program product, characterized in that The method comprises a computer program, which, when executed by a computer, implements the reception behavior recognition method according to any one of claims 1 to 9.

Images