Method, device and electronic equipment for identifying employee reception behavior
By acquiring and analyzing user trajectory and behavioral events, combining multi-level thresholds and virtual trajectory processing, identifying employee reception behaviors, the problem of low recognition accuracy in the existing technology is solved, and is suitable for complex scenarios, improving recognition accuracy and reducing Hardware cost.
Patent Information
- Application Number
- CN202411037849.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-30
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-07-30
Smart Images

Figure CN118570879B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and more particularly to a method, device, electronic device and storage medium for identifying employee reception behavior. The present application also relates to a data processing method, device, electronic device and storage medium. The present application also relates to an employee reception behavior identification system. Background Art
[0002] In the scenario of providing offline retail or other offline services, employee reception of customers is an important business link. Employee reception behavior can often affect the efficiency of business conversion. Identifying employee reception behavior can provide data support for measuring the reception rate and conversion rate of different stores, thereby promoting the improvement of store business conversion. Therefore, how to identify employee reception behavior is particularly important.
[0003] At present, video surveillance is usually used to detect the distance and face-to-face status between employees and customers to determine whether a reception event has occurred. However, since the actual reception scene is often complex, such as dense crowds, which may cause occlusion, there may be abnormal data, making it difficult to ensure the accuracy of reception event detection.
[0004] Therefore, how to improve the accuracy of identifying employee reception behavior is a problem that needs to be solved.
[0005] The above information disclosed in the Background section is only for enhancement of understanding of the background of the present application and therefore it may contain information that does not constitute the prior art that is already known to a person of ordinary skill in the art. Summary of the invention
[0006] An employee reception behavior recognition method provided in an embodiment of the present application solves the problem of low accuracy in employee reception behavior recognition.
[0007] The embodiment of the present application provides an employee reception behavior recognition method, comprising: obtaining sampling data of users in a target area, the sampling data comprising user trajectories, user behavior events, and user features; constructing an employee feature set and its corresponding employee trajectory, and a customer feature set and its corresponding customer trajectory according to the sampling data; recognizing employee reception behavior according to a foot point distance between the employee trajectory and the customer trajectory, and an overlapping field of view angle between the employee and the customer; the overlapping field of view angle is determined based on the employee feature set and the customer feature set, and represents the field of view overlap range between the employee's field of view and the customer's field of view.
[0008] Optionally, the identifying of employee reception behavior based on the footstep distance between the employee trajectory and the customer trajectory, and the overlapping field of view angle between the employee and the customer, includes: when the footstep distance is less than a first distance threshold, the overlapping field of view angle between the employee and the customer satisfies a first overlapping field of view condition and lasts for a period longer than a first period of time threshold, it is determined that employee reception behavior has occurred; and / or, when the footstep distance is greater than the first distance threshold and less than the second distance threshold, the overlapping field of view angle between the employee and the customer satisfies the second overlapping field of view condition and lasts for a period longer than the second period of time threshold, it is determined that employee reception behavior has occurred; and / or, when the footstep distance is greater than the second distance threshold and less than the third distance threshold, the overlapping field of view angle between the employee and the customer satisfies the third overlapping field of view condition and lasts for a period longer than the third period of time threshold, it is determined that employee reception behavior has occurred.
[0009] Optionally, before identifying the employee reception behavior based on the foot point distance between the employee trajectory and the customer trajectory and the overlapping field of view angle between the employee and the customer, it also includes: determining the number of foot points with consistent time in the employee trajectory and the customer trajectory based on the time sorting of the foot points in the user trajectory; eliminating user trajectories whose foot point number is lower than the valid frame number threshold or whose foot point duration corresponding to the user trajectory is lower than the valid frame duration threshold; and using the user trajectories retained after the elimination to determine the foot point distance.
[0010] Optionally, it also includes: when it is determined that the user behavior event corresponding to the customer feature set does not include an entry event, an entry event is supplemented for the first entry trajectory in the customer trajectory corresponding to the customer feature set, so as to recall sampling data of customers who have boarding events or reception events in the target area but whose entry is blocked.
[0011] Optionally, it also includes: obtaining pedestrian re-identification merging results, filtering out single-track users in the target area; the single-track users are users corresponding to unmergeable trajectories; when the user behavior event of the single-track user includes a boarding event, generating a virtual track footpoint for the single-track user, and connecting the virtual track footpoint with the user trajectory of the single-track user after getting off the vehicle in chronological order to form a user trajectory including the virtual track footpoint.
[0012] Optionally, it also includes: if the customer trajectory is a user trajectory including virtual trajectory foot points, when the foot point distance is less than a third distance threshold and the overlapping field of view angle between the employee and the customer meets a third overlapping field of view condition, it is determined that employee reception behavior has occurred.
[0013] Optionally, the method of obtaining sampling data of users in the target area includes: obtaining a video stream captured by a shooting device in the target area; performing feature detection and identification on image frames of the video stream based on a human body detection algorithm model to obtain user features; tracking and detecting the same user according to the user features based on a human body tracking algorithm model to generate a user trajectory of the user; determining the user's orientation and the corresponding user field of view angle according to the user features based on an overlapping field of view algorithm model; and using the user features, user trajectory and user field of view angle as sampling data of the corresponding user.
[0014] Optionally, determining the foot point distance between the employee trajectory and the customer trajectory includes: taking the foot points in the employee trajectory and the customer trajectory as the foot points of the user to be converted; selecting a transformation relationship matrix that is closest to the foot points of the user to be converted from multiple transformation relationship matrices as a target matrix; converting the pixel coordinates of the foot points of the user to be converted into spatial coordinates according to the target matrix; and determining the foot point distance according to the spatial coordinates of the foot points in the employee trajectory and the customer trajectory.
[0015] Optionally, before determining the footstep distance between the employee trajectory and the customer trajectory, it also includes: obtaining the on-site map corresponding to the target area; obtaining a snapshot image captured by a shooting device in the target area; determining multiple conversion units for coordinate mapping between the snapshot image and the on-site map, determining a conversion relationship matrix for each conversion unit, and obtaining multiple conversion relationship matrices; each conversion relationship matrix can be used to convert the pixel coordinates of the user's footsteps to be converted into latitude and longitude coordinates.
[0016] The embodiment of the present application also provides a data processing method, including: obtaining an on-site map corresponding to the target area; obtaining a snapshot image captured by a shooting device in the target area; determining multiple conversion units for coordinate mapping between the snapshot image and the on-site map, and determining a conversion relationship matrix for each of the multiple conversion units to obtain multiple conversion relationship matrices; each conversion relationship matrix can be used to convert the pixel coordinates of the user's foot point to be converted into longitude and latitude coordinates.
[0017] Optionally, the multiple conversion units for determining coordinate mapping between the snapshot image and the on-site map include: dividing the point set of the snapshot image into a triangular mesh by using Delaunay triangulation technology, wherein the circumscribed circle of each triangle does not contain other points in the point set; removing single-sided triangles and unqualified triangles from the multiple triangles of the triangular mesh; wherein the single-sided triangle refers to a triangle with only one side being the boundary edge of the triangular mesh, and the unqualified triangle refers to a triangle with an internal angle exceeding a specific angle threshold; merging the remaining triangles after the elimination, and each merged polygon is a conversion unit, thereby forming multiple conversion units.
[0018] The embodiment of the present application also provides an employee reception behavior recognition device, including: a sampling data unit, used to obtain sampling data of users in a target area, the sampling data including user trajectories, user behavior events and user features; a trajectory construction unit, used to construct an employee feature set and its corresponding employee trajectory, and a customer feature set and its corresponding customer trajectory according to the sampling data; an identification unit, used to identify employee reception behavior according to the foot point distance between the employee trajectory and the customer trajectory, and the overlapping field of view angle between the employee and the customer; the overlapping field of view angle is determined based on the employee feature set and the customer feature set, and characterizes the field of view overlap range between the employee's field of view and the customer's field of view.
[0019] The embodiment of the present application also provides an employee reception behavior recognition system, including: a shooting device set in a target area of employee reception behavior to be identified, an edge server, a cloud microservice cluster, a database, a REID server, and an offline data computing cluster; the edge server is used to obtain a video stream collected by the shooting device in the target area; based on the integrated algorithm model, sampled data of users in the target area are obtained according to the video stream recognition, and the sampled data is provided to the cloud microservice cluster; the sampled data includes user trajectories, user behavior events, and user features; wherein the integrated algorithm model includes a human body detection algorithm model, a human body tracking algorithm model, and an overlapping field of view algorithm model; wherein the human body detection algorithm model is used to obtain user features based on feature detection and recognition of image frames of the video stream; the human body tracking algorithm model is used to track and detect the same user based on user features to generate a user trajectory of the user; the overlapping field of view algorithm model is used to determine the user's direction and the corresponding user field of view angle based on the user features; the cloud microservice cluster is used to obtain the sampled data sent by the edge server and asynchronously store it in the database as sampled data to be merged; the merged sampled data is obtained from the database, and an employee feature set and its corresponding employee trajectory, as well as a customer feature set and its corresponding customer feature set are constructed. corresponding customer trajectory; selecting the transformation relationship matrix closest to the foot points of each user in the user trajectory from multiple transformation relationship matrices as the target matrix, and converting the pixel coordinates of the user foot points into spatial coordinates according to the target matrix; determining the foot point distance between the employee trajectory and the customer trajectory according to the spatial coordinates of each user foot point; identifying the employee reception behavior according to the foot point distance and the overlapping visual angle between the employee and the customer; the overlapping visual angle is determined based on the employee feature set and the customer feature set, and represents the visual field overlap range between the employee's visual field and the customer's visual field; REID server, used to obtain the sampled data to be merged stored in the database , perform pedestrian re-identification merging processing, and store the merging results as merged sampled data in a database; the offline data computing cluster is used to obtain the merged sampled data stored in the database, and filter out the single-track users in the target area; when the user behavior event of the single-track user includes a boarding event, a virtual track footpoint is generated for the single-track user, and the virtual track footpoint is connected with the user track of the single-track user after getting off the car in chronological order to form a user track containing the virtual track footpoint; the single-track user is a user corresponding to an unmergeable track; the database is used to store the sampled data to be merged and the merged sampled data.
[0020] An embodiment of the present application further provides an electronic device, comprising: a memory, and a processor; the memory is used to store a computer program, and after the computer program is run by the processor, the method provided in the embodiment of the present application is executed.
[0021] The embodiment of the present application also provides a computer storage medium storing computer execution instructions, which are used to implement the method provided in the embodiment of the present application when executed by a processor.
[0022] Compared with the prior art, the present application has the following advantages:
[0023] The embodiments of the present application provide a method, device, electronic device and storage medium for identifying employee reception behavior, which obtains sampling data of users in a target area, wherein the sampling data includes user trajectories, user behavior events and user features; constructs an employee feature set and its corresponding employee trajectory, and a customer feature set and its corresponding customer trajectory according to the sampling data; identifies employee reception behavior according to the foot distance between the employee trajectory and the customer trajectory, and the overlapping visual angle between the employee and the customer; the overlapping visual angle is determined based on the employee feature set and the customer feature set, and characterizes the visual overlap range between the employee's visual orientation and the customer's visual orientation. Judging whether reception behavior occurs based on foot distance and overlapping visual orientation is more in line with actual complex reception scenarios, so that employee reception behavior can be more accurately identified, and is particularly suitable for the identification of reception services for larger commodities, such as the identification of reception behavior in car shopping guide scenarios. More accurate employee reception behavior identification can provide a reliable data foundation for smart passenger flow.
[0024] In a preferred method, the coordinate mapping between the snapshot image captured by the shooting device and the on-site map of the target area is divided into multiple conversion units, each of which generates a conversion relationship matrix. When the pixel coordinates of the foot points in the user's trajectory are converted to spatial coordinates, a conversion relationship matrix with the closest distance to the foot points is selected for conversion to provide a more accurate coordinate conversion. The foot point distance between the employee trajectory and the customer trajectory is calculated based on the spatial coordinates, thereby providing a more accurate conversion capability for the calculation of the foot point distance between the employee trajectory and the customer trajectory and solving the image distortion problem of the image snapshot captured by the shooting device to a certain extent, providing a basis for further improving the accuracy of employee reception behavior recognition.
[0025] In another preferred method, when the user behavior event in the target area includes an upload event, a virtual track footpoint is generated for the customer, and the virtual track footpoint is connected with the user track of the single track user after getting off the bus in chronological order. The reception time is extended through the virtual track footpoint, thereby solving the problem of reception behavior recognition in actual offline complex scenarios.
[0026] An employee reception behavior recognition system provided by an embodiment of the present application includes a shooting device, an edge server, a cloud microservice cluster, a database, a REID server, and an offline data computing cluster arranged in a target area of the employee reception behavior to be identified; wherein the edge server integrates an algorithm model, obtains sampling data of users in the target area according to the video stream recognition collected by the shooting device, and provides the sampling data to the cloud microservice cluster; the sampling data includes user trajectories, user behavior events, and user features; the cloud microservice cluster obtains the sampling data sent by the edge server and stores it asynchronously in the database as the sampling data to be merged; obtains the merged sampling data from the database, selects the transformation relationship matrix closest to each user foot point of the user trajectory from multiple transformation relationship matrices as the target matrix, and converts the pixel coordinates of the user foot points of the user trajectory into the target matrix according to the target matrix. Convert to spatial coordinates; determine the foot point distance between the employee trajectory and the customer trajectory according to the spatial coordinates of each user's foot point; identify the employee reception behavior according to the foot point distance and the overlapping visual angle between the employee and the customer; the overlapping visual angle represents the visual overlap range between the employee's visual field and the customer's visual field; the REID server is used to perform pedestrian re-identification and merging processing on the sampled data to be merged, and store the merged results as merged sampled data in the database persistently; the offline data computing cluster is used to obtain the merged sampled data stored in the database, generate virtual track foot points for users whose user behavior events include boarding events, and connect the virtual track foot points with the user trajectory of the single track user after getting off the car in chronological order to form a user trajectory containing virtual track foot points; the database is used to store the sampled data to be merged and the merged sampled data. The system has an elastically scalable architecture, which can make full use of the edge computing capabilities provided by the edge server and the computing resource advantages of each service cluster and server in the cloud, support massive data processing, and thus provide more accurate recognition capabilities and overall solutions for employee reception behaviors, while reducing edge computing power to a certain extent, saving certain hardware costs for stores. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 It is a schematic diagram of a system environment provided by an embodiment of the present application;
[0028] Figure 2 This is a processing flow chart of a method for identifying employee reception behavior provided in the first embodiment of the present application;
[0029] Figure 3 This is a schematic diagram of a process for obtaining sampling data provided by the first embodiment of the present application;
[0030] Figure 4 This is a flowchart of a reception behavior identification method provided in the first embodiment of the present application;
[0031] Figure 5 This is a coordinate conversion diagram of an offline store scenario provided by the first embodiment of the present application;
[0032] Figure 6 is a processing flow chart of a data processing method provided in the first embodiment of the present application;
[0033] Figure 7 This is a schematic diagram of an employee reception behavior recognition device provided in the third embodiment of the present application;
[0034] Figure 8 is a schematic diagram of a data processing device provided in the fourth embodiment of the present application;
[0035] Fig. 9 This is a schematic diagram of the architecture of an employee reception behavior recognition system provided in the fifth embodiment of the present application;
[0036] Fig.10 It is a schematic diagram of an electronic device provided by this application. DETAILED DESCRIPTION
[0037] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.
[0038] The embodiments of the present application provide a method, device, electronic device and storage medium for identifying employee reception behavior. The embodiments of the present application also provide a data processing method, device, electronic device and storage medium. The embodiments of the present application provide a system for identifying employee reception behavior. The following embodiments will describe them one by one.
[0039] To facilitate understanding, we first give the relevant concepts involved in a method for identifying employee reception behavior and the system environment for applying this method.
[0040] Employee reception behavior identification: refers to the use of specific detection and analysis methods to identify whether employees have received customers in offline areas such as stores. Employees include machine employees or real employees, without limitation.
[0041] Footpoints: refers to the footprints or steps of a robot or human projected on the ground. The information of footpoints identified from the video stream includes the coordinates of the footpoints and their corresponding time information.
[0042] Track: A track is a path formed by a series of foot points.
[0043] Event: refers to user behaviors such as entry events / browsing car events in the track.
[0044] Re-identification Merging: It is a technique used in deep learning to handle the task of person re-identification (ReID). Person re-identification refers to finding the same person captured by different cameras by matching and comparing pedestrian images in different camera scenes.
[0045] Please refer to Figure 1, which is an exemplary system environment diagram, including: one or more shooting devices for collecting video streams for the target area, edge servers, cloud microservice clusters, databases, REID servers, and offline data computing clusters. The camera can be a network camera device, such as an IPC (IP Camera, IPC) camera, that is, a camera based on the IP protocol, which can send the collected video data to the server through the communication network. The shooting device (IPC camera) is connected to the Internet of Things platform through triple information. The so-called triple refers to the unique identifier of a device in the Internet of Things, generally including device ID (Device ID), product ID (Product ID) and device key (Device Secret). Among them, the device ID is the unique identifier of the device, the product ID is the unique identifier of the product to which the device belongs, and the device secret key is the key for encrypted communication between the device and the server of the Internet of Things platform. The Internet of Things platform can accurately identify and manage each device through triple information, and ensure the security and reliability of communication between devices. The edge server is located at the logical edge of the network, close to the user side, and provides edge computing. In this application, it especially refers to the edge server of the Internet of Things. The edge server pulls the video stream from the shooting device, identifies and analyzes the video stream according to the integrated algorithm model, uses the processing results as sampled data, and reports the sampled data to the cloud microservice cluster. A communication channel, such as an MQTT channel, an HTTP channel, etc., is established between the edge server and the cloud microservice cluster. The MQTT channel refers to the message queue telemetry transmission (Message Queuing Telemetry Transport, MQTT), for example, reporting sampled data through the MQTT channel; reporting pictures / images through the HTTP channel. The cloud microservice cluster stores the sampled data asynchronously in the database. The cloud microservice cluster identifies the employee reception behavior based on the sampled data. Preferably, since different shooting devices collect video streams, the REID merging server merges the sampled data to be merged in the database. The cloud microservice cluster can also identify the employee reception behavior based on the merged sampled data. Considering the complexity of the actual offline environment, for example, for scenes of large-scale commodities such as automobiles or scenes with large passenger flow, occlusion is prone to occur, so the sampled data in the database is processed by the offline data computing cluster, such as removing abnormal data and completing data. The employee reception behavior recognition deployed in the system environment is an important part of the smart customer flow scenario. The accuracy of reception behavior recognition directly affects the accuracy of key indicators such as the store's reception timeliness rate and reception conversion rate in the smart customer flow.
[0046] It can be understood that the system environment is only an embodiment of the application scenario of the employee reception behavior identification method. The purpose of providing this scenario embodiment is to facilitate the understanding of the employee reception behavior identification method provided in this application, and is not used to limit the method provided in this application.
[0047] It should be noted that the above disclosed information is only used to help understand the present application and does not mean to constitute the prior art known to ordinary technicians in the field.
[0048] An employee reception behavior recognition method provided in an embodiment of the present application constructs an employee feature set and its corresponding employee trajectory, as well as a customer feature set and its corresponding customer trajectory based on sampling data of a target area; identifies employee reception behavior according to the foot point distance between the employee trajectory and the customer trajectory, and the overlapping field of view angle between the employee and the customer, which is more in line with actual complex reception scenes, and can accurately identify employee reception behavior for complex scenes. The target area is further divided into multiple conversion units, and a corresponding conversion relationship matrix is provided for each conversion unit, thereby realizing a more accurate coordinate conversion rule, making the foot point distance calculation more accurate, and improving the accuracy of reception behavior recognition.
[0049] The technical solution shown in the present application is described in detail below through specific embodiments. It can be understood that the following embodiments can exist independently or in combination with each other, and the same or similar contents will not be described repeatedly in different embodiments.
[0050] The following combination Figures 2 to 5 The employee reception behavior recognition method provided in the first embodiment of the present application is described. Figure 2 The employee reception behavior identification method shown includes:
[0051] Step S201, obtaining sample data of users in a target area, wherein the sample data includes user trajectories, user behavior events, and user characteristics;
[0052] Step S202, constructing an employee feature set and its corresponding employee trajectory, and a customer feature set and its corresponding customer trajectory according to the sampled data;
[0053] Step S203, identifying the employee reception behavior according to the footstep distance between the employee trajectory and the customer trajectory, and the overlapping visual angle between the employee and the customer; the overlapping visual angle is determined based on the employee feature set and the customer feature set, and represents the visual overlap range between the employee's visual direction and the customer's visual direction.
[0054] The execution subject of the embodiment of the present application can be an electronic device, or an employee reception behavior recognition device in the electronic device. The employee behavior recognition device can be a virtual device implemented by software, or a device implemented by a combination of software and hardware. The electronic device can specifically be a cloud server such as a cloud microservice cluster, or an edge server, or even a terminal in a store, etc., without specific limitation.
[0055] In this embodiment, the target area may refer to an area for video shooting and / or image acquisition in offline retail or other offline service scenarios, such as a product display area of a store, a store front desk, etc. The sampled data refers to the sampled data obtained by identifying and analyzing the video stream (i.e., video data) collected by the shooting device. The shooting device is used to shoot video and collect images of the target area. The shooting device can be an IPC camera. The video data collected can be provided to an electronic device such as an edge server in the form of a video stream through a communication network. The edge server identifies and analyzes the video stream to obtain the sampled data. Users can be humans in the video stream, including employees and customers, and of course, can also include robot employees. Based on a variety of algorithm models, user data is calculated and sampled from the video stream, such as foot sampling / human feature sampling / event behavior sampling / head feature sampling, etc., to obtain sampled data. The sampled data can specifically include user trajectories, user behavior events, and user features. The user trajectory can be a moving path composed of user foot points, which can include employee trajectories and customer trajectories. User behavior events refer to events that characterize specific behaviors of users in the target area, such as user entry events, user exit events, product browsing events (such as car browsing events), product experience events (such as car boarding events), etc. User features include feature information corresponding to the user, such as body features, head features, orientation features (such as body orientation and / or facial orientation), etc. Body features may include body postures, such as car boarding postures, etc.; head features may include head mark features, etc. User features may also include other features, which are not specifically limited.
[0056] As described in step S201, the sampling data of users in the target area is obtained. Specifically, the execution subject can pull the video stream for analysis and processing to obtain the sampling data, or obtain the sampling data obtained by other electronic devices in advance based on the video stream analysis and processing. In one embodiment, the edge server can pull the video stream of the shooting device for processing to obtain the sampling data. Specifically, the acquisition of the sampling data of users in the target area includes: obtaining the video stream collected by the shooting device in the target area; performing feature detection and identification on the image frames of the video stream based on the human body detection algorithm model to obtain user features; tracking and detecting the same user based on the user features based on the human body tracking algorithm model to generate the user trajectory of the user; determining the user's direction and the corresponding user's field of view angle based on the user features based on the overlapping field of view algorithm model; using the user features, user trajectory and user field of view angle as the sampling data of the corresponding user. In this way, the edge computing capability provided by the edge server can be fully utilized to calculate the sampling data of the user. Among them, the human body detection algorithm model can be a pre-set human body detection model, which is used to perform feature detection on the image frames of the video stream to identify the user features and their corresponding quality scores, and the quality score refers to the quality evaluation score corresponding to the user features. Generally speaking, the clearer the user image part in the image frame of the video stream, the higher the quality score of the user feature; the clarity of the user image part contained in the image frame is affected by light brightness, occlusion, etc. Among them, the human body tracking algorithm model is an algorithm model that tracks and detects the same user based on user features, and is used to generate user trajectory footpoints. Among them, the overlapping field of view algorithm model is an algorithm model that calculates the user orientation (including body orientation and / or facial orientation), for example, the facial orientation is determined based on the characteristics of the human eye position information. For another example, the body orientation is determined based on the body position and posture. Furthermore, in subsequent steps, the respective field of view (field of view angle) can be determined based on the user's orientation, so as to determine whether the user's field of view overlaps. For example, when the overlapping part of the employee and customer's respective fields of view exceeds a certain field of view angle threshold, it is judged that there is an overlapping field of view.
[0057] Furthermore, the acquisition of sampling data of users in the target area also includes: determining the foot points of the users in each image frame of the video stream; wherein the information of the foot points includes the coordinates of the foot points and the time information; and determining the user trajectory corresponding to the same user according to the foot points. The user's crossing line event, i.e., the entry event of entering the target area or the departure event of leaving the target area, can be determined according to the positional relationship between the user trajectory and the line segments corresponding to the entrance and exit of the target area. The vehicle can be directly identified according to the algorithm model and whether the user has a boarding event. The customer's experience event (such as a boarding event) can also be determined according to the identified human body posture characteristics of the customer and the positional relationship between the customer trajectory and the commodity (such as a car). When the edge server undertakes the calculation of a large number of foot points and cross-line relationships, the computing pressure of the cloud microservice cluster can be reduced. Of course, the head point information can also be used instead of the foot point information to form the user trajectory. The head point information is based on the coordinate information of the user's head identified in the video stream and its corresponding time information.
[0058] Please refer to Figure 3 , the figure shows a schematic process of obtaining sampling data, including:
[0059] S301, pull the video stream from the shooting device, and analyze each frame of the video stream. S302, call the human detection model to perform human detection. S303, extract user features and calculate the quality score. S304, determine whether each frame of the picture is the same user based on the user features, and track the foot / head trajectory. S305, calculate the user's crossing events (entry / exit events) during the tracking process. S306, after the tracking is completed, extract the user features with the highest quality score, and use the user features, user trajectories, user behavior events, etc. as sampling data. The process of acquiring sampling data can be executed by an electronic device that provides edge computing capabilities, and of course, the cloud microservice cluster can directly pull the video stream to execute the process.
[0060] As described in step S202, an employee feature set and its corresponding employee trajectory, as well as a customer feature set and its corresponding customer trajectory are constructed based on the sampled data. Specifically, employees and customers are identified based on the sampled data, and then employee feature sets and their corresponding employee trajectories, as well as customer feature sets and their corresponding customer trajectories are constructed based on the identification results. Among them, employees can be identified in a variety of ways, which are not specifically limited in this application. For example, employee feature data can be identified through historical trajectories and historical credible feature recognition (such as head features), and employees are identified based on the similarity between the sampled data and the learned employee feature data. After the employees are identified, the employee's sampled data and the customer's sampled data can be distinguished.
[0061] This step also includes data filtering and noise reduction processing to improve the accuracy of the recognition results. Specifically, it includes: determining the number of foot points that are consistent in time in the employee trajectory and the customer trajectory based on the time sorting of the foot points in the user trajectory; eliminating user trajectories whose foot point number is lower than the effective frame number threshold or whose foot point duration corresponding to the user trajectory is lower than the effective frame duration threshold; and using the user trajectory retained after the elimination to determine the foot point distance. Therefore, in the subsequent steps, when the employee trajectory and the customer trajectory are compared for intersection (including comparison based on the distance or position of the foot points and calculation of the field of view angle), the effective frame numbers of the two trajectories can be compared, that is, the compared effective frame numbers have time consistency.
[0062] Preferably, it also includes filtering out single-track users and split personnel in the venue without user behavior events, and processing them specially, so as to improve the accuracy of reception behavior recognition. In actual implementation, the REID server can use REID technology to merge the sampled data to be merged in the database, and the unmergeable track is a single track; the sampled data to be merged and the merged sampled data in the database can be processed offline through the offline data computing cluster, including: for users entering the venue, if there is no data of the entry event, the first entry track is supplemented with an event based on whether the user is an employee or a customer, so as to avoid the situation where the entry is blocked during the event recall and is not recalled due to the lack of entry events. Among them, the processing related to the single-track user specifically includes: obtaining the pedestrian re-identification merging results, filtering out the single-track users in the target area; the single-track user is a user corresponding to an unmergeable track; when the user behavior event of the single-track user includes a boarding event, a virtual track footpoint is generated for the single-track user, and the virtual track footpoint is connected with the user track after the single-track user gets off the bus in chronological order to form a user track containing the virtual track footpoint. In actual implementation, it can also include: adding tags to the user track containing the virtual track footpoint and / or its corresponding user features, so as to facilitate the selection of appropriate footpoint distance threshold / overlapping field of view condition / duration threshold for such user track in subsequent steps to identify employee reception behavior. In this way, the complex scene of offline reception can be met. For example, after a short reception time outside the car, the customer and the employee are talking in the car in the blind spot of the camera. According to the boarding event, the virtual track footpoint can continue the track when the track disappears (boarding), and the reception time can be continued, thereby providing a more accurate reception time and a more accurate reception number.
[0063] Among them, the processing related to the split personnel in the venue includes: when it is determined that the user behavior event corresponding to the customer feature set does not include an entry event, the entry event is supplemented for the first entry trajectory in the customer trajectory corresponding to the customer feature set, so as to recall the sampling data of customers who have boarding events or reception events in the target area but whose entry is blocked.
[0064] As described in step S203, the employee's reception behavior is identified based on the footpoint distance between the employee's trajectory and the customer's trajectory, and the overlapping field of view angle between the employee and the customer. In an actual reception scenario, the employee and the customer can be face-to-face, side-by-side, or at a certain angle, and there is a certain intersection or overlap range in their fields of view, that is, they are looking at the same position. Therefore, it is more reasonable to judge whether a reception behavior occurs based on the footpoint distance and the overlapping field of view angle.
[0065] After distinguishing the respective sampling data of the employee and the customer in the above steps, the respective field of view ranges (field of view angles) can be further determined according to the orientations of the employee and the customer, so as to judge whether the fields of view of the employee and the customer overlap.
[0066] During implementation, multi-level distance thresholds of the footpoints and / or multi-level overlapping field of view conditions (or multi-level overlapping field of view angle thresholds) can be designed to achieve more accurate judgment of the reception behavior. Further, the reception behavior can be comprehensively identified in combination with the duration (or stay duration). Specifically, the identifying the employee's reception behavior according to the footpoint distance between the employee's trajectory and the customer's trajectory, and the overlapping field of view angle between the employee and the customer includes: when the footpoint distance is less than the first distance threshold (A1), the overlapping field of view angle between the employee and the customer meets the first overlapping field of view condition (C1), and the duration is greater than the first duration threshold (B1), it is determined that an employee reception behavior occurs; and / or, when the footpoint distance is greater than the first distance threshold (A1) and less than the second distance threshold (A2), the overlapping field of view angle between the employee and the customer meets the second overlapping field of view condition (C2), and the duration is greater than the second duration threshold (B2), it is determined that an employee reception behavior occurs; and / or, when the footpoint distance is greater than the second distance threshold (A2) and less than the third distance threshold (A3), the overlapping field of view angle between the employee and the customer meets the third overlapping field of view condition (C3), and the duration is greater than the third duration threshold (B3), it is determined that an employee reception behavior occurs. Among them, the distance thresholds of the footpoints satisfy A1 < A2 < A3, forming multi-level thresholds. The duration threshold B2 is a low threshold, B2 is a medium threshold, and B3 is a high threshold. The overlapping field of view condition can be an overlapping field of view angle threshold, C1 < C2 < C3, so that when the distance between the employee and the customer is close enough, the judgment condition for the overlapping field of view angle is relatively wide. For example, when the employee and the customer are standing side by side and the footpoint distance (the distance based on the actual spatial position) is less than 0.5 meters, the overlapping range of the orientations of the employee and the customer does not need to be too large to perform normal reception behavior.
[0067] In this embodiment, if the customer trajectory is a user trajectory including virtual trajectory foot points, when the foot point distance is less than the third distance threshold (A3), and the overlapping visual angle between the employee and the customer meets the third overlapping visual angle condition (C3), it is determined that the employee reception behavior occurs. That is, when the customer does not have an entry / exit event, but has a boarding event, a larger distance threshold and a wider overlapping visual angle condition are set to identify the employee reception behavior.
[0068] Please refer to Figure 4 , the figure shows a schematic process of reception behavior recognition, including:
[0069] S401, construct employee sets and customer sets based on sampled data. According to employee and customer identification results, construct employee sets and their corresponding foot point data, and construct customer sets and their corresponding foot point data. If a customer has a boarding event, a virtual track foot point is generated. At the same time, if the customer does not have an entry or exit event but has other events such as a boarding event, a larger reception time threshold (i.e., duration threshold) is set. S402, perform data filtering and noise reduction processing, including: sorting the foot points according to the time, comparing the effective frame numbers of the two trajectories (time consistency), and filtering directly if the effective frame number of the foot points is too short (time threshold filtering); and then calculating the foot point distance between the foot points. It also includes: filtering out single trajectories and split personnel in the venue without events, and performing special processing to improve the accuracy of recognition. S403, generate virtual track foot points for boarding events / experience events of single-track users. S404, compare the intersection of employee trajectories and customer trajectories, and determine whether an employee reception event occurs according to the multi-level distance threshold / overlapping field condition rules of the foot points. Including: If the foot point distance threshold is less than the first threshold A1 and there is an overlapping field of view C1 and the reception time meets the low threshold B1, it is judged as reception. If the foot point distance threshold is greater than the first threshold A1 and less than the second threshold A2 and there is an overlapping field of view angle C2 and the reception time meets the middle threshold B2, it is judged as reception. If the foot point distance threshold is greater than the second threshold A2 and less than the high threshold A3 and there is an overlapping field of view angle C3 and the reception time meets the high threshold B3, it is judged as reception. If it is a virtual track foot point, the foot point distance threshold is less than the threshold A3 and there is an overlapping field of view angle C3, it is judged as reception. S405, the analysis results are recorded in the database. S406, offline data processing, calculation of reception rate / conversion rate / average reception time and other business indicators.
[0070] In this embodiment, the foot point distance is a distance calculated based on the spatial position, that is, the position distance calculated after the pixel coordinates of the foot points of the employee trajectory and the customer trajectory are converted into spatial coordinates (i.e., longitude and latitude coordinates). The pixel coordinates of the foot point refer to the pixel coordinates of the foot point contained in the sampling data in the picture collected by the shooting device, and the spatial coordinates refer to the coordinate information converted from the pixel coordinates into the world coordinate system of the on-site map, that is, the actual position information in the real world. Coordinate transformation is performed through a homography matrix. Preferably, the digital spatial recognition capability of the target area is constructed through a multi-level transformation relationship matrix. Specifically, by dividing a plurality of transformation units, each transformation unit corresponds to a transformation relationship matrix (i.e., corresponds to a homography matrix), thereby forming a multi-level transformation relationship matrix, from which the transformation relationship matrix closest to the foot point to be converted is selected, and the pixel coordinates of the foot point are converted into longitude and latitude coordinates using the matrix. Wherein, the determination of the foot point distance between the employee trajectory and the customer trajectory includes: taking the foot points in the employee trajectory and the customer trajectory as the foot points of the user to be converted; selecting the conversion relationship matrix with the closest distance to the foot points of the user to be converted from multiple conversion relationship matrices as the target matrix; converting the pixel coordinates of the foot points of the user to be converted into spatial coordinates according to the target matrix; and determining the foot point distance according to the spatial coordinates of the foot points in the employee trajectory and the customer trajectory. Wherein, before determining the foot point distance between the employee trajectory and the customer trajectory, it also includes determining a multi-level conversion relationship matrix from an image plane to a world plane, specifically: obtaining an on-site map corresponding to the target area; obtaining a snapshot image captured by a shooting device in the target area; determining multiple conversion units for coordinate mapping between the snapshot image and the on-site map, determining a conversion relationship matrix for each conversion unit, and obtaining multiple conversion relationship matrices; each conversion relationship matrix can be used to convert the pixel coordinates of the foot points of the user to be converted into longitude and latitude coordinates.
[0071] Preferably, the image plane is divided into multiple transformation units by triangulation technology, and a multi-level transformation relationship matrix is constructed, that is, the transformation relationship is generated based on local points, and the distance between the actual point to be transformed and different matrices is used to determine which transformation relationship matrix to use for coordinate transformation, rather than performing overall coordinate mapping, so as to achieve more accurate distance calculation. When the snapshot image pulled by the shooting device is distorted, for example: the lines on the normal ground in the actual scene are all straight lines, but the snapshot pulled by the camera may be a curve with a certain curvature. A more reasonable transformation relationship matrix can be selected by triangulation. The transformation relationship matrix is a homography matrix, which is a 3x3 transformation matrix in two-dimensional space. Specifically, it includes the following processing: (1) Find the key points on the image through Delaunay triangulation, that is, divide the point set into a triangular grid, and the circumscribed circle of each triangle does not contain other points in the point set; (2) Perform triangle screening and mark single-edge triangles (markSingleEdgeTriangles) and unqualified triangles (markIllTriangles). Among them, a single-sided triangle refers to a triangle in which only one side is the boundary edge of the triangle mesh, and an unqualified triangle refers to a triangle whose internal angle exceeds a certain threshold. (3) Merge triangles (mergeTriangle), merge suitable triangles to form one or more polygons, each polygon is a basic unit of transformation relationship, that is, transformation unit. For example, merge triangles with internal angles below a certain threshold or merge acute triangles. (4) Create a homography matrix for each merged polygon. It is usually completed by performing a perspective transformation on the matching relationship between the source point and the target point. It includes: finding the corresponding points of the vertices of each polygon in the source image and the target image; using these matching point pairs, calling the functions of graphics libraries such as OpenCV to calculate the homography matrix. Usually, at least four pairs of matching points are required to calculate a certain homography matrix. The homography matrix represents the geometric transformation from the polygon in the source image to the corresponding polygon in the target image. For example, the homography matrix H can transform the point P (x, y) in the source image to the point P' (x', y') in the target image, using homogeneous coordinates, and the transformation relationship is P' = H*P. (5) These homography matrices are decoded into an array of length 9, since a 3x3 matrix contains 9 elements; they are packaged into a JSON object and can be output as a string.
[0072] Please refer to Figure 5, the figure shows a schematic coordinate transformation of an offline store scene. For the checkpoints in the image (i.e., the foot points to be transformed), a multi-level transformation relationship matrix between pixel coordinates and spatial coordinates is determined based on multiple calibration points. Then, based on the transformation relationship matrix (homography matrix) corresponding to the transformation unit closest to the checkpoint and the pixel coordinates of the checkpoint, the spatial coordinates (i.e., longitude and latitude coordinates) of the checkpoint are calculated to achieve the transformation between pixel coordinates and real-world coordinates.
[0073] In this embodiment, judging whether reception behavior occurs based on the foot point distance, the overlapping field of view toward the direction, and the duration (i.e., reception time) is more in line with the actual reception scene, and the judgment condition is more reasonable. In addition, a multi-level conversion relationship matrix is used in the calculation of the foot point distance (which can be understood as the reception distance), which can provide more accurate conversion capabilities and solve the problem of camera image distortion to a certain extent. Furthermore, the virtual track foot points generated based on the boarding event (i.e., the experience event) extend the reception time, thereby being able to meet the reception behavior recognition requirements of actual complex reception scenes.
[0074] It should be noted that, in the absence of conflict, the features given in this embodiment and other embodiments of the present application can be combined with each other, and steps S201 and S202 or similar terms do not limit the steps to be executed sequentially.
[0075] So far, the method provided in this embodiment has been explained. In the method, whether reception behavior occurs is judged based on the foot distance and the overlapping direction field of view between the employee trajectory and the customer trajectory. This method is more in line with the actual complex reception scenes, so that employee reception behavior can be more accurately identified, and is particularly suitable for the identification of larger commodity reception services, such as the reception behavior identification in car shopping guide scenarios.
[0076] Based on the above embodiment, the second embodiment of the present application provides a data processing method. Figure 6 The method is described below. For the same parts, please refer to the description of the corresponding parts of the above embodiments and will not be repeated here. Figure 6 The data processing method shown comprises:
[0077] Step S601, obtaining an on-site map corresponding to the target area;
[0078] Step S602, obtaining a snapshot image captured by a shooting device in the target area;
[0079] Step S603, determine multiple conversion units for coordinate mapping between the snapshot image and the on-site map, determine a conversion relationship matrix for each of the multiple conversion units, and obtain multiple conversion relationship matrices; each conversion relationship matrix can be used to convert the pixel coordinates of the user's foot point to be converted into longitude and latitude coordinates.
[0080] In this embodiment, the image plane is divided into multiple conversion units by triangulation technology, and a multi-level conversion relationship matrix is constructed, that is, the conversion relationship is generated based on local points. Among them, the multiple conversion units for determining the coordinate mapping between the snapshot image and the on-site map include: dividing the point set of the snapshot image into a triangular mesh by Delaunay triangulation technology, wherein the circumscribed circle of each triangle does not contain other points in the point set; removing single-sided triangles and unqualified triangles from multiple triangles of the triangular mesh; wherein the single-sided triangle refers to a triangle with only one side being the boundary side of the triangular mesh, and the unqualified triangle refers to a triangle with an internal angle exceeding a specific angle threshold; merging the remaining triangles after the elimination, and each merged polygon is a conversion unit, thereby forming multiple conversion units. The step of determining the transformation relationship matrix for each of the multiple transformation units to obtain multiple transformation relationship matrices includes: determining corresponding points of the polygon vertices of each transformation unit in the snapshot image and the on-site map to form matching point pairs; wherein the snapshot image is a source image of the transformation relationship, and the on-site map is a target image of the transformation relationship; using the matching point pairs to perform perspective transformation to obtain a homography matrix corresponding to each transformation unit as the transformation relationship matrix of the transformation unit. That is, each polygon vertex is used as a calibration point of the transformation unit.
[0081] It should be noted that the method is not limited to application in employee reception behavior recognition scenarios, but can also be applied to any other scenarios involving conversion of pixel coordinates to space coordinates.
[0082] So far, the method provided by this embodiment has been described. In the method, the coordinate mapping between the snapshot image captured by the shooting device and the on-site map of the target area is divided into multiple conversion units, each conversion unit generates a conversion relationship matrix, and when the pixel coordinates of the foot points in the user's trajectory are converted into spatial coordinates, a conversion relationship matrix with the closest distance to the foot points is selected for conversion to provide a more accurate coordinate conversion. The foot point distance between the employee trajectory and the customer trajectory is calculated based on the spatial coordinates, thereby providing a more accurate conversion capability for the calculation of the foot point distance between the employee trajectory and the customer trajectory and solving the image distortion problem of the image snapshot captured by the shooting device to a certain extent, providing a basis for further improving the accuracy of employee reception behavior recognition.
[0083] Corresponding to the first embodiment, the third embodiment of the present application provides an employee reception behavior recognition device. For the relevant parts, please refer to the description of the corresponding method embodiment. Figure 7 , the employee reception behavior recognition device shown in the figure includes:
[0084] A sampling data unit 701 is used to obtain sampling data of users in a target area, wherein the sampling data includes user trajectories, user behavior events, and user characteristics;
[0085] A trajectory construction unit 702, configured to construct an employee feature set and its corresponding employee trajectory, and a customer feature set and its corresponding customer trajectory according to the sampled data;
[0086] The identification unit 703 is used to identify the employee reception behavior according to the footstep distance between the employee trajectory and the customer trajectory, and the overlapping field of view angle between the employee and the customer; the overlapping field of view angle is determined based on the employee feature set and the customer feature set, and represents the field of view overlap range between the employee's field of view and the customer's field of view.
[0087] Optionally, the identification unit 703 is specifically used to: when the foot point distance is less than a first distance threshold, the overlapping field of view angle between the employee and the customer meets the first overlapping field of view condition and the duration is greater than the first duration threshold, it is determined that the employee reception behavior has occurred; and / or, when the foot point distance is greater than the first distance threshold and less than the second distance threshold, the overlapping field of view angle between the employee and the customer meets the second overlapping field of view condition and the duration is greater than the second duration threshold, it is determined that the employee reception behavior has occurred; and / or, when the foot point distance is greater than the second distance threshold and less than the third distance threshold, the overlapping field of view angle between the employee and the customer meets the third overlapping field of view condition and the duration is greater than the third duration threshold, it is determined that the employee reception behavior has occurred.
[0088] Optionally, the trajectory construction unit 702 is specifically used to: determine the number of footsteps in the employee trajectory and the customer trajectory that are consistent in time according to the time sorting of the footsteps in the user trajectory; eliminate user trajectories whose footstep numbers are lower than the valid frame number threshold or whose footstep durations corresponding to the user trajectories are lower than the valid frame duration threshold; and use the user trajectories retained after the elimination to determine the footstep distance.
[0089] Optionally, the trajectory construction unit 702 is specifically used to: when it is determined that the user behavior event corresponding to the customer feature set does not include an entry event, supplement the entry event with the first entry trajectory in the customer trajectory corresponding to the customer feature set, so as to recall the sampling data of customers who have boarding events or reception events in the target area but whose entry is blocked.
[0090] Optionally, the trajectory construction unit 702 is specifically used to: obtain pedestrian re-identification merging results, filter out single-trajectory users in the target area; the single-trajectory users are users corresponding to unmergeable trajectories; when the user behavior event of the single-trajectory user includes a boarding event, generate a virtual trajectory footpoint for the single-trajectory user, and connect the virtual trajectory footpoint with the user trajectory of the single-trajectory user after getting off the vehicle in chronological order to form a user trajectory including the virtual trajectory footpoint.
[0091] Optionally, the identification unit 703 is specifically used to: if the customer trajectory is a user trajectory including virtual trajectory foot points, then when the foot point distance is less than a third distance threshold and the overlapping field of view angle between the employee and the customer meets a third overlapping field of view condition, it is determined that employee reception behavior has occurred.
[0092] Optionally, the sampling data unit 701 is specifically used to: obtain a video stream captured by a shooting device in a target area; perform feature detection and recognition on image frames of the video stream based on a human body detection algorithm model to obtain user features; track and detect the same user according to the user features based on a human body tracking algorithm model to generate a user trajectory of the user; determine the user's orientation and the corresponding user's field of view angle according to the user features based on an overlapping field of view algorithm model; and use the user features, user trajectory and user field of view angle as sampling data for the corresponding user.
[0093] Optionally, the device also includes a coordinate conversion unit, which is used to: use the foot points in the employee trajectory and the customer trajectory as the foot points of the user to be converted; select the transformation relationship matrix that is closest to the foot points of the user to be converted from multiple transformation relationship matrices as the target matrix; convert the pixel coordinates of the foot points of the user to be converted into spatial coordinates according to the target matrix; determine the foot point distance according to the spatial coordinates of the foot points in the employee trajectory and the customer trajectory.
[0094] Optionally, the coordinate conversion unit is further used to: obtain the on-site map corresponding to the target area; obtain a snapshot image captured by a shooting device in the target area; determine multiple conversion units for coordinate mapping between the snapshot image and the on-site map, and determine a conversion relationship matrix for each conversion unit to obtain multiple conversion relationship matrices; each conversion relationship matrix can be used to convert the pixel coordinates of the user's foot point to be converted into latitude and longitude coordinates.
[0095] Corresponding to the second embodiment, the fourth embodiment of the present application provides a data processing device. For the relevant parts, please refer to the description of the corresponding method embodiment. Figure 8 , the data processing device shown in the figure comprises:
[0096] A map importing unit 801 is used to obtain an on-site map corresponding to the target area;
[0097] An image acquisition unit 802 is used to acquire a snapshot image captured by a shooting device in a target area;
[0098] The conversion relationship determination unit 803 is used to determine multiple conversion units for coordinate mapping between the snapshot image and the on-site map, and determine a conversion relationship matrix for each of the multiple conversion units to obtain multiple conversion relationship matrices; each conversion relationship matrix can be used to convert the pixel coordinates of the user's foot point to be converted into longitude and latitude coordinates.
[0099] Optionally, the conversion relationship determination unit 803 is specifically used to: divide the point set of the snapshot image into a triangular mesh by using the Delaunay triangulation technology, wherein the circumscribed circle of each triangle does not contain other points in the point set; remove single-sided triangles and unqualified triangles from multiple triangles of the triangular mesh; wherein the single-sided triangle refers to a triangle with only one side being the boundary edge of the triangular mesh, and the unqualified triangle refers to a triangle with an internal angle exceeding a specific angle threshold; merge the remaining triangles after the elimination, and each merged polygon is a conversion unit, thereby forming multiple conversion units.
[0100] Based on the above embodiments, the fifth embodiment of the present application provides an employee reception behavior recognition system. Figure 1 The system is described below. For the same parts, please refer to the description of the corresponding parts of the above embodiments and will not be repeated here. Figure 1 The employee reception behavior recognition system shown includes: a shooting device 101 arranged in a target area of the employee reception behavior to be recognized, an edge server 102, a cloud microservice cluster 103, a database 104, a REID server 105, and an offline data computing cluster 106;
[0101] The edge server is used to obtain the video stream collected by the shooting device in the target area; based on the integrated algorithm model, the sampling data of users in the target area are obtained according to the video stream recognition, and the sampling data is provided to the cloud microservice cluster; the sampling data includes user trajectory, user behavior events and user characteristics; wherein, the integrated algorithm model includes a human body detection algorithm model, a human body tracking algorithm model and an overlapping field of view algorithm model; wherein, the human body detection algorithm model is used to obtain user characteristics based on feature detection and recognition of image frames of the video stream; the human body tracking algorithm model is used to track and detect the same user based on the user characteristics to generate the user trajectory of the user; the overlapping field of view algorithm model is used to determine the user's direction and the corresponding user field of view angle based on the user characteristics;
[0102] The cloud microservice cluster is used to obtain the sample data sent by the edge server and asynchronously store it in the database as sample data to be merged; obtain the merged sample data from the database, construct an employee feature set and its corresponding employee trajectory, and a customer feature set and its corresponding customer trajectory; select a conversion relationship matrix that is closest to the distance between each user's foot point in the user trajectory from multiple conversion relationship matrices as a target matrix, and convert the pixel coordinates of the user's foot point into spatial coordinates according to the target matrix; determine the foot point distance between the employee trajectory and the customer trajectory according to the spatial coordinates of each user's foot point; identify the employee reception behavior according to the foot point distance and the overlapping visual angle between the employee and the customer; the overlapping visual angle is determined based on the employee feature set and the customer feature set, and represents the visual field overlap range between the employee's visual field and the customer's visual field;
[0103] The REID server is used to obtain the sampled data to be merged stored in the database, perform pedestrian re-identification merging processing, and store the merging results as merged sampled data in the database persistently;
[0104] The offline data computing cluster is used to obtain the merged sampled data stored in the database, and filter out the single-track users in the target area; when the user behavior event of the single-track user includes a boarding event, a virtual track footpoint is generated for the single-track user, and the virtual track footpoint is connected with the user track of the single-track user after getting off the car in chronological order to form a user track including the virtual track footpoint; the single-track user is a user corresponding to an unmergeable track;
[0105] The database is used to store the sample data to be merged and the sample data that has been merged.
[0106] In this embodiment, a communication channel is established between the edge server and the cloud microservice group so that the sampling data and pictures / images are reported to the cloud microservice cluster, such as establishing an MQTT / HTTP channel. The sampling data reported by the edge server to the cloud microservice cluster specifically includes but is not limited to: foot points / human body sampling / event behavior sampling / head feature sampling, etc. The cloud microservice cluster asynchronously records the reported sampling data to the database. Furthermore, the system also includes a gateway cluster. The gateway cluster receives the message request from the edge server and forwards it to the cloud microservice cluster. Furthermore, the system also includes an MQ (Message queue, MQ) middleware cluster, that is, a message queue middleware cluster. The message sent by the edge server to the cloud server cluster is written to the MQ middleware, asynchronously buffered by the MQ middleware, and then forwarded to the server-side microservice cluster for processing.
[0107] Furthermore, the offline data computing cluster also conducts indicator analysis, such as calculating business indicators such as reception timeliness rate, conversion rate (customers are converted from potential users to actual users), and average reception time.
[0108] The system provided in this embodiment has the characteristics of elastic and scalable architecture, and provides a complete solution from the video stream collection of offline actual reception places (such as stores), employee reception behavior recognition, offline indicator calculation, etc., making full use of edge computing capabilities and cloud service computing resource capabilities, reducing edge computing power and saving certain hardware costs. Through the edge, the cross-line relationship between a large number of foot points and the entrance and exit line segments of the target area is calculated to judge the user entry / exit events, reduce the computing pressure of the cloud microservice cluster, and use the data processing capabilities of the REID server to merge the trajectory REID, so as to achieve fast data calculation. In calculating the distance between the foot points of the trajectories, a local conversion unit based on multiple calibration points is selected for coordinate conversion, so as to obtain the actual spatial position of the user in the target area, so that the user trajectory and the relative spatial position between different trajectories can be more accurately identified. Further, the judgment can be assisted by setting a multi-level threshold to accurately identify the reception behavior of employees.
[0109] Please refer to Fig. 9 , the figure shows a schematic employee reception behavior recognition system architecture, where 0.0 to 0.3 can be understood as the preparatory steps for obtaining sampled data to ensure the accuracy and validity of the data. 1.1 to 1.10 are the calculation and processing of the sampled data. 0.0: The shooting device (IPC camera) is connected to the Internet of Things platform through triple login registration to build a communication channel. 0.1: The cloud microservice cluster can issue configurations through the MQTT / HTTP channel, such as for on-site calibration information, field digitization information (map coordinate information), etc. 0.2: The edge computing server can report data to the cloud microservice cluster for processing through the MQTT / HTTP channel, such as MQTT reporting sampling data, HTTP uploading pictures, etc. 0.3: After receiving the request, the gateway can forward it to the cloud microservice cluster for processing.
[0110] 1.1: The edge server pulls the stream from the camera, performs calculations, calls the algorithm model for recognition and adds analysis and calculation logic. 1.2: Report various sampling data through the MQTT channel, such as foot points / body sampling / event behavior sampling / head feature sampling, etc. 1.3: Perform asynchronous buffering of services through the MQ middleware and forward it to the cloud microservice cluster for processing. 1.4 to 1.6: The cloud microservice cluster processes the data and asynchronously stores the sampled data for REID merging. 1.7 to 1.8: REID pulls the sampled data, performs REID merging, and stores the data persistently. 1.9 to 1.10: The offline data computing cluster performs some data analysis and synchronizes the results to the database.
[0111] So far, the system provided in this embodiment has been explained. The system has an elastically scalable architecture, can fully utilize the edge computing capabilities provided by the edge server and the computing resource advantages of various service clusters and servers in the cloud, support massive data processing, and thus provide more accurate recognition capabilities and overall solutions for reception behaviors. At the same time, it reduces edge computing power to a certain extent, saving certain hardware costs for stores.
[0112] Based on the above embodiment, one embodiment of the present application provides an electronic device. For the relevant parts, please refer to the corresponding description of the above embodiment. Fig.10 Schematic diagram of an electronic device, the electronic device shown in the figure includes: a memory, and a processor; the memory is used to store a computer program, and after the computer program is run by the processor, the method provided in the embodiment of the present application is executed.
[0113] Based on the above embodiments, one embodiment of the present application provides a computer storage medium. For the relevant parts, please refer to the corresponding description of the above embodiments. The schematic diagram of the computer storage medium is similar to the schematic diagram of an electronic device, and the memory in the figure can be understood as the storage medium. The computer storage medium stores computer execution instructions, which are used to implement the method provided in the embodiment of the present application when executed by the processor.
[0114] It should be noted that the embodiments of this application may involve the use of user data. In actual applications, the user's specific personal data should be used in the scheme described herein within the scope permitted by applicable laws and regulations in accordance with the applicable laws and regulations of the country (for example, the user's explicit consent, the user's effective notification, etc.). The collection of relevant data of personnel in this application has obtained the permission and consent of the parties concerned.
[0115] In a typical configuration, an electronic device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory. The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.
[0116] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.
[0117] Although the present application is disclosed as above in the form of a preferred embodiment, it is not intended to limit the present application. Any technical personnel in this field may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.
Claims
1. A method for identifying employee reception behavior, characterized in that: include: Acquiring sampling data of users in a target area, including: acquiring a video stream captured by a shooting device in the target area; obtaining sampling data of users in the target area according to the video stream recognition based on an integrated algorithm model, and asynchronously storing the sampling data as sampling data to be merged; wherein the sampling data includes user trajectories, user behavior events, and user features; wherein the integrated algorithm model includes a human body detection algorithm model, a human body tracking algorithm model, and an overlapping field of view algorithm model; wherein the human body detection algorithm model is used to obtain user features based on feature detection and recognition of image frames of the video stream; the human body tracking algorithm model is used to track and detect the same user based on user features to generate a user trajectory of the user; the overlapping field of view algorithm model is used to determine the user's orientation and the corresponding user field of view angle based on the user features; Obtain the stored sampled data to be merged for pedestrian re-identification merging processing, and use the merging result as the merged sampled data; Obtain merged sampled data, construct employee feature sets and their corresponding employee trajectories, as well as customer feature sets and their corresponding customer trajectories; A transformation relationship matrix closest to each user's foot point in the user trajectory is selected from multiple transformation relationship matrices as a target matrix, and pixel coordinates of the user's foot points are converted into spatial coordinates according to the target matrix; the foot point distance between the employee trajectory and the customer trajectory is determined according to the spatial coordinates of each user's foot point; the employee reception behavior is identified according to the foot point distance between the employee trajectory and the customer trajectory, and the overlapping visual angle between the employee and the customer; the overlapping visual angle is determined based on the employee feature set and the customer feature set, and represents the visual field overlap range between the employee's visual field and the customer's visual field; The method further includes: filtering out single-track users in the target area according to the merged sampled data; when the user behavior event of the single-track user includes a boarding event, generating a virtual track footpoint for the single-track user, and connecting the virtual track footpoint with the user track after the single-track user gets off the bus in chronological order to form a user track including the virtual track footpoint; the single-track user is a user corresponding to an unmergeable track.
2. The method according to claim 1, characterized in that The identifying of the employee reception behavior according to the footstep distance between the employee trajectory and the customer trajectory, and the overlapping visual angle between the employee and the customer, includes: When the foot distance is less than the first distance threshold, the overlapping visual angle between the employee and the customer meets the first overlapping visual condition and the duration is greater than the first duration threshold, it is determined that the employee reception behavior occurs; and / or, When the foot distance is greater than the first distance threshold and less than the second distance threshold, the overlapping visual angle between the employee and the customer meets the second overlapping visual condition, and the continuous waiting time is greater than the second time threshold, it is determined that the employee reception behavior occurs; and / or, When the foot point distance is greater than the second distance threshold and less than the third distance threshold, the overlapping field of view angle between the employee and the customer meets the third overlapping field of view condition and the duration is greater than the third duration threshold, it is determined that the employee reception behavior has occurred.
3. The method according to claim 2, characterized in that Before identifying the employee reception behavior according to the footstep distance between the employee trajectory and the customer trajectory and the overlapping visual angle between the employee and the customer, the method further includes: Determine the number of points with the same time in the employee trajectory and the customer trajectory according to the time order of the points in the user trajectory; Eliminate user trajectories whose number of footsteps is lower than the effective frame number threshold or whose footstep duration corresponding to the user trajectory is lower than the effective frame duration threshold; The user trajectory retained after the elimination is used to determine the foot point distance.
4. The method according to claim 1, characterized in that Also includes: When it is determined that the user behavior event corresponding to the customer feature set does not include an entry event, an entry event is supplemented for the first entry trajectory in the customer trajectory corresponding to the customer feature set, so as to recall the sampling data of customers who have boarding events or reception events in the target area but whose entry is blocked.
5. The method according to claim 1, characterized in that Also includes: If the customer trajectory is a user trajectory including virtual trajectory foot points, when the foot point distance is less than the third distance threshold and the overlapping field of view angle between the employee and the customer meets the third overlapping field of view condition, it is determined that the employee reception behavior occurs.
6. The method according to claim 1, characterized in that Before determining the foot point distance between the employee trajectory and the customer trajectory, the method further includes: Obtaining an on-site map corresponding to the target area; Obtaining snapshot images captured by a shooting device in the target area; Determine multiple conversion units for coordinate mapping between the snapshot image and the on-site map, determine a conversion relationship matrix for each conversion unit, and obtain multiple conversion relationship matrices; each conversion relationship matrix can be used to convert the pixel coordinates of the user's foot point to be converted into longitude and latitude coordinates.
7. The method according to claim 6, characterized in that The multiple conversion units for determining coordinate mapping between the snapshot image and the on-site map include: Dividing the point set of the snapshot image into a triangular mesh by using a Delaunay triangulation technique, wherein the circumscribed circle of each triangle does not contain other points in the point set; Eliminate single-sided triangles and unqualified triangles from the multiple triangles of the triangular mesh; wherein the single-sided triangle refers to a triangle with only one side being a boundary edge of the triangular mesh, and the unqualified triangle refers to a triangle with an internal angle exceeding a specific angle threshold; The remaining triangles after culling are merged, and each merged polygon is a conversion unit, thereby forming a plurality of conversion units.
8. A device for identifying employee reception behavior, characterized in that: include: A sampling data unit, used to obtain sampling data of users in a target area, including: obtaining a video stream captured by a shooting device in the target area; obtaining sampling data of users in the target area according to the video stream recognition based on an integrated algorithm model, and storing the sampling data asynchronously as sampling data to be merged; wherein the sampling data includes user trajectories, user behavior events and user features; wherein the integrated algorithm model includes a human body detection algorithm model, a human body tracking algorithm model and an overlapping field of view algorithm model; wherein the human body detection algorithm model is used to obtain user features based on feature detection and recognition of image frames of the video stream; the human body tracking algorithm model is used to track and detect the same user based on user features to generate a user trajectory of the user; the overlapping field of view algorithm model is used to determine the user's orientation and the corresponding user field of view angle according to the user features; A trajectory construction unit, used to obtain the merged sampled data, construct an employee feature set and its corresponding employee trajectory, and a customer feature set and its corresponding customer trajectory; an identification unit, configured to identify the reception behavior of the employee according to the footstep distance between the employee trajectory and the customer trajectory, and the overlapping visual angle between the employee and the customer; the overlapping visual angle is determined based on the employee feature set and the customer feature set, and represents the visual overlap range between the employee's visual direction and the customer's visual direction; The device is also used to: obtain the stored sampled data to be merged to perform pedestrian re-identification merging processing, and use the merging result as the merged sampled data; The device is also used to: select a conversion relationship matrix closest to each user's foot point in the user trajectory from a plurality of conversion relationship matrices as a target matrix, convert the pixel coordinates of the user's foot points into spatial coordinates according to the target matrix; determine the foot point distance between the employee trajectory and the customer trajectory according to the spatial coordinates of each user's foot point; The trajectory construction unit is further used to: filter out single-trajectory users in the target area according to the merged sampling data; when the user behavior event of the single-trajectory user includes a boarding event, generate a virtual trajectory footpoint for the single-trajectory user, and connect the virtual trajectory footpoint with the user trajectory of the single-trajectory user after getting off the vehicle in chronological order to form a user trajectory including the virtual trajectory footpoint; the single-trajectory user is a user corresponding to an unmergeable trajectory.
9. An employee reception behavior recognition system, characterized in that: include: The camera equipment, edge server, cloud microservice cluster, database, REID server, and offline data computing cluster are set up in the target area where the employee reception behavior is to be identified; The edge server is used to obtain the video stream collected by the shooting device in the target area; based on the integrated algorithm model, the sampling data of users in the target area are obtained according to the video stream recognition, and the sampling data is provided to the cloud microservice cluster; the sampling data includes user trajectory, user behavior events and user characteristics; wherein the integrated algorithm model includes a human body detection algorithm model, a human body tracking algorithm model and an overlapping field of view algorithm model; wherein the human body detection algorithm model is used to obtain user characteristics based on feature detection and recognition of image frames of the video stream; the human body tracking algorithm model is used to track and detect the same user based on user characteristics to generate the user trajectory of the user; the overlapping field of view algorithm model is used to determine the user's direction and the corresponding user field of view angle based on user characteristics; The cloud microservice cluster is used to obtain the sampled data sent by the edge server and asynchronously store it in the database as sampled data to be merged; obtain the merged sampled data from the database, construct an employee feature set and its corresponding employee trajectory, and a customer feature set and its corresponding customer trajectory; select a conversion relationship matrix that is closest to the foot points of each user in the user trajectory from multiple conversion relationship matrices as a target matrix, and convert the pixel coordinates of the user foot points into spatial coordinates according to the target matrix; determine the foot point distance between the employee trajectory and the customer trajectory according to the spatial coordinates of each user foot point; identify the employee reception behavior according to the foot point distance and the overlapping visual angle between the employee and the customer; the overlapping visual angle is determined based on the employee feature set and the customer feature set, and represents the visual field overlap range between the employee's visual field and the customer's visual field; The REID server is used to obtain the sampled data to be merged stored in the database, perform pedestrian re-identification merging processing, and store the merging results as merged sampled data in the database persistently; The offline data computing cluster is used to obtain the merged sampled data stored in the database, and filter out the single-track users in the target area; when the user behavior event of the single-track user includes a boarding event, a virtual track footpoint is generated for the single-track user, and the virtual track footpoint is connected with the user track of the single-track user after getting off the car in chronological order to form a user track including the virtual track footpoint; the single-track user is a user corresponding to an unmergeable track; The database is used to store the sample data to be merged and the sample data that has been merged.
10. An electronic device, characterized in that: include: A memory and a processor; the memory is used to store a computer program, and after the computer program is run by the processor, the method described in any one of claims 1 to 7 is executed.
