Video processing, display and completion method, device, system and storage medium

Abstract

The embodiment of the present application provides a video processing, display and completion method, device, system and storage medium. In the embodiment of the present application, for the video data captured by multiple cameras in the target space, these video data are merged and processed based on the movable objects entering the target space to obtain the trajectory of each movable object, and the trajectory of each movable object is associated with the data object related to its behavior in the target space, which provides conditions for video processing based on the movable object as a unit and is conducive to simplifying the video processing operation. Furthermore, in the process of information display, not only the trajectory of the movable object can be displayed, but also the data object related to its behavior in the target space can be associated and displayed, and the relevant information of the movable object in the target space can be displayed in a multi-dimensional and stereoscopic manner, which can improve the efficiency of video processing and reduce the cost of video processing.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a video processing, display and completion method, device, system and storage medium. Background Art

[0002] In real life, cameras are deployed in many scenarios for security monitoring. For example, in offline unmanned retail settings, multiple cameras are often deployed to monitor customer shopping behavior. Another example is in homes, where cameras are installed in different areas such as the living room, bedroom, and kitchen to monitor the activities of elderly people and children at home to ensure their safety.

[0003] In camera surveillance scenarios, if a specific situation occurs, such as a missed payment or an elderly person or child falling, the system typically uses video footage from all cameras on site to conduct investigations. This video surveillance backtracking method is costly and inefficient. Summary of the Invention

[0004] Various aspects of the present application provide a video processing, display and completion method, device, system and storage medium to simplify video-based troubleshooting operations, improve troubleshooting efficiency and reduce troubleshooting costs.

[0005] An embodiment of the present application provides a video processing method, including: obtaining video data captured by multiple cameras in a target space, wherein the target space contains at least one movable object; merging the video data captured by the multiple cameras in units of movable objects to obtain a trajectory of the at least one movable object; and associating the trajectory of the at least one movable object with at least one data object, wherein the at least one data object is related to the behavior of the movable object in the target space.

[0006] An embodiment of the present application also provides a trajectory display method, comprising: in response to a first query operation, sending a first data request to a server device to request a target trajectory; the target trajectory is the trajectory of a target movable object within a target space; receiving the target trajectory and at least one data object associated with the target trajectory returned by the server device; and during the target trajectory display process, displaying the at least one data object in association with the at least one data object, wherein the at least one data object is related to the behavior of the target movable object within the target space.

[0007] An embodiment of the present application also provides a trajectory completion method, including: displaying a target trajectory, where the target trajectory is the trajectory of a target movable object in a target space, and the target trajectory has a missing portion; determining, based on the direction of the target movable object before the missing portion, a candidate camera in the target space that is likely to have captured the missing portion; and completing the target trajectory based on video content corresponding to the missing time period captured by the candidate camera; the missing time period is the time period corresponding to the missing portion.

[0008] An embodiment of the present application provides a video processing system, comprising: multiple cameras, a server device, and a display terminal deployed in a target space; the target space allows movable objects to enter and exit; the multiple cameras are used to capture video data within their respective fields of view and upload the captured video data to the server device; the server device is used to merge the video data of the multiple cameras in units of movable objects to obtain the trajectory of each movable object entering the target space, and associate the trajectory of each movable object with at least one data object, wherein the at least one data object is related to the behavior of the movable object in the target space; the display terminal is used to obtain the target trajectory and at least one data object associated with it from the server device according to a query operation, and in the process of displaying the target trajectory, associate and display the at least one data object associated with it; the target trajectory is the trajectory of the target movable object in the target space.

[0009] A server device provided in an embodiment of the present application includes: a memory and a processor; the memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program to: obtain video data captured by multiple cameras in a target space, where the target space contains at least one movable object; merge the video data captured by the multiple cameras in units of movable objects to obtain the trajectory of the at least one movable object; and associate the trajectory of the at least one movable object with at least one data object, where the at least one data object is related to the behavior of the movable object in the target space.

[0010] An embodiment of the present application provides a display terminal, comprising: a memory, a processor, a communication component, and a display; the memory is configured to store a computer program; the processor, coupled to the memory, is configured to execute the computer program, and is configured to: in response to a first query operation, send a first data request to a server device to request a target trajectory; the target trajectory is a trajectory of a target movable object within a target space; receive the target trajectory returned by the server device and at least one data object associated with the target trajectory; and during the target trajectory display process, associate and display the at least one data object, wherein the at least one data object is related to the behavior of the target movable object within the target space.

[0011] An embodiment of the present application also provides a display terminal, comprising: a memory, a processor, a communication component, and a display; the memory is used to store a computer program; the processor is coupled to the memory, and is used to execute the computer program, so as to: display a target trajectory, wherein the target trajectory is the trajectory of a target movable object in a target space, and there is a missing portion in the target trajectory; based on the direction of the target movable object before the missing portion, determining a candidate camera in the target space that is likely to capture the missing portion; and completing the target trajectory based on the video content corresponding to the missing time period captured by the candidate camera; the missing time period is the time period corresponding to the missing portion.

[0012] An embodiment of the present application provides a computer-readable storage medium storing a computer program. When the computer program is executed by one or more processors, the one or more processors implement the steps of the video processing method, trajectory display method, or trajectory completion method provided in the embodiment of the present application.

[0013] In an embodiment of the present application, the video data captured by multiple cameras within the target space is merged and processed based on the movable objects entering the target space to obtain the trajectory of each movable object. The trajectory of each movable object is then associated with the data objects related to its behavior in the target space. This provides conditions for video processing based on the movable object as a unit, which is conducive to simplifying video processing operations. Furthermore, during the information display process, not only can the trajectory of the movable object be displayed, but the data objects related to its behavior in the target space can also be associated and displayed. The relevant information of the movable object in the target space can be displayed in a multi-dimensional and stereoscopic manner, which can improve video processing efficiency and reduce video processing costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0015] Figure 1 A schematic structural diagram of a video processing system provided by an exemplary embodiment of the present application;

[0016] Figure 2 A schematic diagram of the structure of a video processing system applied to offline stores provided by an exemplary embodiment of the present application;

[0017] Figure 3a A schematic diagram of a first query interface provided by an exemplary embodiment of the present application;

[0018] Figure 3b Another schematic diagram of a first query interface provided by an exemplary embodiment of the present application;

[0019] Figure 4a-4d A schematic diagram of the state of a display interface provided by an exemplary embodiment of the present application;

[0020] Figure 5 A schematic diagram showing camera orientation on a store map provided for an exemplary embodiment of the present application;

[0021] Figure 6a A schematic diagram of adding marking information related to a second behavior in a track provided by an exemplary embodiment of the present application;

[0022] Figure 6b A schematic diagram of a second query interface provided by an exemplary embodiment of the present application;

[0023] Figure 7a A flowchart of a video processing method provided by an exemplary embodiment of the present application;

[0024] Figure 7b A flowchart of another video processing method provided by an exemplary embodiment of the present application;

[0025] Figure 8 A schematic flow chart of a trajectory display method provided by an exemplary embodiment of the present application;

[0026] Figure 9 A schematic flow chart of a trajectory completion method provided by an exemplary embodiment of the present application;

[0027] Figure 10a A schematic diagram of the structure of a server device provided by an exemplary embodiment of the present application;

[0028] Figure 10bA schematic structural diagram of a display terminal provided by an exemplary embodiment of the present application;

[0029] Figure 10c A schematic structural diagram of another display terminal provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0030] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0031] In response to the technical problems of high cost and low efficiency faced by existing video screening scenarios, in an embodiment of the present application, the video data captured by multiple cameras in the target space are merged and processed based on the movable objects entering the target space to obtain the trajectory of each movable object, and the trajectory of each movable object is associated with the data object related to its behavior in the target space, which provides conditions for trajectory processing (such as screening, editing or beautification) based on the movable object, and is conducive to simplifying the trajectory processing (such as screening, editing or beautification) operation. Furthermore, in the information display process, not only the trajectory of the movable object can be displayed, but also the data object related to its behavior in the target space can be associated and displayed, and the relevant information of the movable object in the target space can be displayed in three dimensions from multiple directions, which can improve the efficiency of trajectory processing and reduce the cost of trajectory processing.

[0032] The following describes in detail the technical solutions provided by various embodiments of the present application in conjunction with the accompanying drawings.

[0033] Figure 1 This is a schematic diagram of the structure of a video processing system provided by an exemplary embodiment of the present application. Figure 1 As shown, the system includes: multiple cameras 11 deployed in the target space 10, a server device 12 and a display device 13. The server device 12 is communicatively connected with the multiple cameras 11 and the display terminal 13 respectively.

[0034] In this embodiment, the target space 10 refers to a physical space with accommodation capacity that allows movable objects 14 to enter and exit. Movable objects 14 generally refer to any movable object, including autonomously movable objects such as users, robots, or unmanned vehicles; and non-autonomously movable objects such as vehicles driven by humans. Depending on the application scenario, the specific implementation of the target space 10 will vary, and accordingly, the movable objects that can enter and exit the target space 10 will also vary. The following examples illustrate:

[0035] In some application scenarios, the target space 10 is an offline store such as a shopping mall or supermarket. Accordingly, the movable objects are users entering and exiting the offline store. The users here include consumers and / or supermarket staff, or may also include autonomous shopping carts that can move freely in offline stores, or robots, etc.

[0036] In other application scenarios, the target space 10 is a home space, and accordingly, the movable objects are users who can enter and exit the home space. The users here include family members and / or external visitors, or may also include home service robots that can move freely in the home environment, such as autonomously movable purifiers, sweeping robots, etc.

[0037] In some other application scenarios, the target space 10 is a public place such as a bus station, a railway station, or an airport. Accordingly, the movable objects include passengers who can enter and exit these places, venue staff, delivery staff, people who pick up and drop off passengers, taxi drivers and other personnel, and may also include private cars, taxis, etc.

[0038] Regardless of the application scenario, there are always some movable objects entering and exiting the target space 10. In order to meet the monitoring needs, multiple cameras 11 can be installed in the target space 10. Multiple cameras 11 are installed at different positions in the target space 10, with different fields of view, and can capture video data within their respective fields of view. Optionally, depending on the monitoring needs and the intensity of the monitoring, the coverage of the target space 10 by the multiple cameras 11 may be different. Preferably, the fields of view of the multiple cameras 11 can overlap or seamlessly connect with each other, so as to achieve comprehensive and seamless coverage of the target space 10, but it is not limited to this.

[0039] In this embodiment, the type of camera 11 is not limited. For example, in terms of signal transmission, camera 11 can be an analog camera or a digital camera. In terms of image quality, camera 11 can be a standard definition camera or a high definition camera. In terms of appearance, camera 11 can be a spherical camera, a hemispherical camera, or a gun-style camera. In terms of viewing angle, camera 11 can be a wide-angle camera or a standard camera. In terms of the number of lenses, camera 11 can be a monocular camera or a binocular camera.

[0040] In this embodiment, the multiple cameras 11 can upload captured video data to the server device 12 via a communication connection between the multiple cameras 11 and the server device 12. Optionally, the multiple cameras 11 can upload the captured video data to the server device 12 in real time, or according to a specific upload strategy, such as periodically or at a fixed time. The video data captured by each camera 11 includes movable objects entering its field of view, the related behavior of the movable objects within the target space 10, and the time, trajectory, and position of the movable objects within the target space 10.

[0041] In this embodiment, the implementation form of the server device 12 is not limited, and it can be a server device such as a conventional server, a cloud server or a server array. The server device 12 is mainly responsible for receiving the video data uploaded by multiple cameras 11 and saving these video data so that these video data can be further processed later. Furthermore, the server device 12 is also used to: merge the video data shot by multiple cameras 11 in units of movable objects to obtain the trajectory of each movable object entering the target space 10. Among them, the trajectory of the movable object mainly describes the activity trajectory of the movable object in the target space; in terms of implementation form, the trajectory of the movable object can be a video (referred to as trajectory video for short), or a GIF (referred to as trajectory GIF for short), or an image picture (referred to as trajectory picture for short). Among them, one trajectory for each movable object is conducive to processing related to the trajectory of the movable object in units of movable objects. Optionally, if the trajectory of the movable object is a trajectory video, the processing related to the trajectory of the movable object can be video processing, and the video processing here includes but is not limited to: video screening, video editing or video beautification, etc., without the need to screen the video content captured by all cameras to obtain a video of a movable object, which can improve video processing efficiency and reduce video processing costs. Preferably, in each embodiment of the present application, the trajectory of the movable object is a trajectory video. Figure 1 、 Figure 2 、 Figures 4a-6a In the figure, trajectory video is used as an example for illustration, but it is not limited to this.

[0042] Furthermore, the server device 12 can also associate the trajectory of each movable object with at least one data object. Among them, at least one data object is related to the behavior of the movable object in the target space. In other words, these data objects can reflect the behavior of the movable object in the target space to a certain extent. In this embodiment, the data objects used are not limited. Depending on the application scenario, target space and movable objects, the data objects that can reflect the behavior of the movable object in the target space will also be different. The following examples are provided:

[0043] For example, in some application scenarios, the target space is an offline store, and the movable objects are users entering and exiting the offline store. In offline retail scenarios, offline stores typically offer a variety of items for users to purchase, such as fresh produce, daily necessities, clothing, cosmetics, home appliances, and electronics. User behavior within offline stores primarily includes entering and exiting the store, walking around the store, browsing items, selecting items, purchasing items, and paying at checkout. Data objects that can reflect user behavior within offline stores include, but are not limited to, the time users enter and exit the store, details of their in-store orders, and / or maps of offline stores. The time users enter and exit the offline store can be used to determine the time users enter and exit the store and the length of time they spend in the store. The details of users' in-store orders can be used to determine the items they purchased, the items they paid for, the time they paid, and the POS machine used for payment. Through the offline store map, we can know the internal layout and aisle location of the offline store, and then know the approximate location of users walking, browsing items, selecting items, purchasing items, and paying and checking out in the offline store.

[0044] For example, in other application scenarios, the target space is a home space, and the movable objects are family members. In a home scenario, the home space typically includes household appliances for family members to use, such as televisions, robot vacuums, microwave ovens, air conditioners, and lights, as well as food and beverages. Family members' behavior within the home space primarily includes entering and exiting the home space, using appliances for chores or recreational activities, such as watching TV, turning on the air conditioner or lights, cooking on the stove, and heating food in the microwave. Data objects that reflect family members' behavior within the home environment include, but are not limited to, the time family members enter and exit the home space, appliance usage time, remaining food and beverage levels, and / or a map of the home environment. The time family members enter and exit offline stores can be used to determine the time family members enter and exit the home space and the length of time they spend there. Detailed information within the home environment, such as appliance usage time and remaining food and beverage levels, can be used to determine what family members do within the home space and when they do these things. Through the map of the home space, you can know the location of various home appliances, and then you can know the approximate location of family members when they are doing various things in the home environment.

[0045] For example, in yet other application scenarios, the target space is a public place such as a bus station, train station, or airport, and movable objects include passengers entering and exiting these places. In public settings, these places typically include various service windows, such as ticket sales / refund windows and baggage check-in windows, as well as self-service equipment such as ticket machines and gates. Passengers' activities within these public places primarily include purchasing / refunding tickets, collecting tickets, checking in luggage, and passing through gates. Data objects that can reflect passenger behavior within public places include, but are not limited to, the time a passenger enters and exits the public place, information about the service performed at the service window, information about passenger operations on self-service equipment, and / or maps of the public place. The time a passenger enters and exits a public place can be used to determine the time the passenger enters and exits the public place and the length of time the passenger remains within the public place. Information about the service performed at the service window and information about passenger operations on self-service equipment can be used to determine what the passenger does within the public place and when they do it. Through the map of public places, we can know the location of service windows, self-service equipment, etc. in public places, and then we can know the approximate location of tourists when they do various things in public places.

[0046] In combination with the specific implementation of the data objects listed in the above-mentioned multiple scenario embodiments, in an optional embodiment, the data objects that can reflect the behavior of the movable object in the target space can be roughly summarized into the following categories: the first category: time data of the movable object entering and exiting the target space; the second category: detailed behavior data of the movable object in the target space; the third category: map data of the target space.

[0047] In this optional embodiment, at least one of the three types of data objects described above can be used. That is, the server device 12 can associate the trajectory of the movable object with the time data of the movable object entering and exiting the target space, detailed data on the movable object's behavior within the target space, and / or map data of the target space, but the present invention is not limited thereto. Furthermore, the specific data objects used can also be flexibly set based on monitoring requirements.

[0048] After obtaining the trajectory of a movable object and at least one associated data object, this data can be used as a data basis to process the trajectory of any movable object according to video processing needs, such as screening, editing, or beautification. For ease of description and distinction, in this embodiment of the application, the movable object that needs to be processed related to its trajectory is referred to as the target movable object. The target movable object can be any movable object that enters the target space. In this embodiment, the trajectory of the target movable object can be processed by the display terminal 13.

[0049] Specifically, the display terminal 13 can obtain the target trajectory and at least one data object associated therewith from the server device 12 according to the query operation, wherein the target trajectory refers to the trajectory of the target movable object in the target space. Afterwards, the display terminal 13 displays the target trajectory, and in the process of displaying the target trajectory, displays at least one data object associated with the target trajectory. Optionally, if the target trajectory is a trajectory video, the display terminal 13 can play the trajectory video, and in the process of playing the trajectory video, displays at least one data object associated with the trajectory video. In this way, relevant processing personnel can understand the target trajectory and its related content, and then process the trajectory of the target movable object. For example, in a video investigation scenario, based on understanding the trajectory of the target movable object and at least one data object associated therewith, it is possible to investigate whether the target movable object has abnormal behavior or specified behavior.

[0050] Among them, the trajectory of at least one movable object is associated with at least one data object that can reflect the behavior of the movable object in the target space. During the trajectory-related processing, while displaying the trajectory, other information related to the trajectory can also be displayed in multiple dimensions, which is conducive to multi-dimensional considerations, improving the trajectory-related processing efficiency (improving video screening efficiency) and saving costs.

[0051] In the embodiment of the present application, the server device 12 needs to perform at least: merging processing and association operation. Among them, merging processing is an abbreviation for "merging the video data captured by multiple cameras 11 in units of movable objects to obtain the trajectory of each movable object entering the target space"; association operation is an abbreviation for "associating the trajectory of each movable object with at least one data object". In the embodiment of the present application, the detailed implementation methods of "merging processing" and "association operation" are not limited, and any implementation method that can achieve the above purpose is applicable to the embodiment of the present application. The detailed implementation methods of "merging processing" and "association operation" are exemplified below.

[0052] An exemplary description of the implementation method of "merge processing":

[0053] In some exemplary embodiments of the present application, the server device 12 uses a combination of computer vision (CV) technology and person re-identification (REID) technology to merge and process the video data captured by multiple cameras 11, thereby obtaining the trajectory of each movable object entering the target space. In the embodiments of the present application, CV technology is primarily used to track the movable objects captured by each camera 11; REID technology is primarily used to identify whether the video data captured by one camera contains a movable object captured by another camera.

[0054] Specifically, the server device 12 can perform target tracking on the video data captured by each of the multiple cameras 11 to obtain the movable objects captured by each of the multiple cameras 11; and perform REID processing on the movable objects captured by each of the multiple cameras 11 to obtain the trajectory of at least one movable object. For cameras 11 that capture the same movable object, the trajectory segments containing the movable object captured by these cameras 11 can be spliced ​​together based on the time sequence in which these cameras captured the movable object to obtain the trajectory of the movable object.

[0055] The target tracking of the video data captured by the plurality of cameras 11 can be achieved in a variety of ways, which are not limited to this. The following examples illustrate:

[0056] In option a1, target tracking is performed once per frame of video data for each camera. This means that for each frame captured by the camera, the system determines whether the current frame contains a movable object tracked in the previous frame. This approach provides finer tracking granularity, reduces the probability of missed tracking, and more comprehensively tracks movable objects within the target space.

[0057] In optional method a2, for each camera, a target tracking operation is performed for every N frames of video data, that is, in each target tracking operation, N frames of video data are combined to determine whether the movable object tracked last time is tracked; wherein N is an integer greater than or equal to 2. In this optional embodiment, instead of tracking and classifying the two frames of video data, a delayed confirmation technology is used to improve the accuracy of the tracking results, taking into account issues such as body occlusion, clothing similarity, and light reflection. Delayed confirmation technology means that instead of making a real-time judgment on each frame of video data, a judgment is made on 2 or more frames of video data, because after the delayed judgment, there will be more and more complete trajectory information (for example, more faces, full-body photos, etc.), which is conducive to improving the accuracy of tracking and classification.

[0058] The above-mentioned REID processing of the movable objects captured by the multiple cameras 11 can be implemented in a variety of ways, which are not limited to this. The following examples illustrate:

[0059] In optional manner b1, REID processing is performed on the movable objects captured by each of the multiple cameras 11 in combination with the overlapping fields of view of the multiple cameras 11 to obtain the trajectory of at least one movable object.

[0060] In optional manner b2, REID processing is performed on the movable objects captured by each of the multiple cameras 11 in combination with the temporal and spatial information of the movable objects to obtain the trajectory of at least one movable object.

[0061] In optional manner b3, REID processing is performed on the movable objects captured by each of the multiple cameras 11 in combination with feature information of the movable objects obtained by other means to obtain a trajectory of at least one movable object.

[0062] In optional manner b4, REID processing is performed on the movable objects captured by each of the multiple cameras 11 in combination with the overlapping fields of view of the multiple cameras 11 and the spatiotemporal information of the movable objects to obtain the trajectory of at least one movable object.

[0063] In optional method b5, combined with the overlapping fields of view between multiple cameras 11 and the feature information of the movable objects obtained by other means, REID processing is performed on the movable objects photographed by each of the multiple cameras 11 to obtain the trajectory of at least one movable object.

[0064] In optional method b6, the movable objects captured by multiple cameras 11 are processed by REID based on the temporal and spatial information of the movable objects and the feature information of the movable objects obtained by other means to obtain the trajectory of at least one movable object.

[0065] In optional method b7, the overlapping fields of view between multiple cameras 11, the spatiotemporal information of the movable objects, and the feature information of the movable objects obtained by other means are combined, and REID processing is performed on the movable objects photographed by each of the multiple cameras 11 to obtain the trajectory of at least one movable object.

[0066] In the above-mentioned optional embodiment, by utilizing the overlapping fields of view of the cameras, some less clear movable objects can be associated based on the positions of the movable objects captured by different cameras, thereby improving tracking precision and accuracy and making the trajectory of the same movable object more complete. By utilizing the spatiotemporal information of the movable objects to filter the associations between the movable objects, associations between movable objects that clearly do not meet the spatiotemporal requirements can be deleted, reducing false merging and improving the accuracy of the trajectory. Spatiotemporal information refers to information obtained by establishing a coordinate system for a map of the target space and making reasonable judgments about the location of the movable object at which time. By utilizing the feature information of the movable object obtained by other means and combining it with the features of the movable object captured by the camera for comprehensive merging, it is beneficial to improve the accuracy of merging, so that a trajectory is composed of trajectory segments of the same movable object as much as possible. Optionally, if the trajectory is a video trajectory, the trajectory segments are video segments captured by the camera.

[0067] Among them, depending on the application scenario, other implementation methods for obtaining the characteristic information of the movable object and the characteristic information obtained will also be different. Taking offline stores as an example, obtaining the characteristic information of the movable object by other means includes at least one of the following: 1. Obtaining the facial image of the movable object (mainly referring to the user who consumes in the store) taken by the POS machine in the offline store; 2. Obtaining the facial image in the electronic account corresponding to the movable object. The electronic account refers to the account registered by the movable object online, and the facial image in the account refers to the facial image of the movable object bound to the account during the registration process.

[0068] An exemplary description of the implementation of "association operation":

[0069] Depending on the application scenario, target space, and movable object, the behavior of the movable object within the target space will vary. Accordingly, the at least one data object that can reflect the movable object's behavior within the target space will also vary. Depending on the data object, the implementation method for associating the movable object's trajectory with the at least one data object will also vary. In the following embodiments, the implementation method of the association operation is exemplified by the example of at least one data object including: time data of the movable object entering and exiting the target space, detailed data on the movable object's behavior within the target space, and / or map data of the target space.

[0070] Among them, for different movable objects, the implementation process of associating their trajectories with data objects is the same or similar. In the following embodiment, the first movable object is taken as an example for explanation. The first movable object can be any movable object that enters and exits the target space.

[0071] In optional manner c1, the trajectory of the first movable object may be associated with time data of the first movable object entering and exiting the target space.

[0072] In optional manner c2, the trajectory of the first movable object may be associated with detailed behavior data of the first movable object in the target space.

[0073] In alternative manner c3, the trajectory of the first movable object may be associated with map data of the target space.

[0074] In optional manner c4, the trajectory of the first movable object may be associated with the time data of the first movable object entering and exiting the target space and the behavior detail data of the first movable object in the target space.

[0075] In optional manner c5, the trajectory of the first movable object may be associated with time data of the first movable object entering and exiting the target space and map data of the target space.

[0076] In optional manner c6, the trajectory of the first movable object may be associated with the behavior detail data of the first movable object in the target space and the map data of the target space.

[0077] In optional manner c7, the trajectory of the first movable object may be associated with time data of the first movable object entering and exiting the target space, detailed behavior data of the first movable object in the target space, and map data of the target space.

[0078] Regardless of the optional method, an optional implementation method for associating the trajectory of the first movable object with the time data of the first movable object entering and exiting the target space includes: generating timeline information for the trajectory of the first movable object based on the time data of the first movable object entering and exiting the target space. That is, the time data of the first movable object entering and exiting the target space is reflected on the timeline of the trajectory of the first movable object. For example, the start time of the timeline can be set to the time when the first movable object enters the target space, and the end time of the timeline can be set to the time when the first movable object leaves the target space.

[0079] Regardless of which optional method is used, an optional implementation method of associating the trajectory of the first movable object with the map data of the target space includes: adding a dynamic icon to the map data of the target space based on the trajectory of the first movable object, and the dynamic icon is linked to the first movable object. In other words, as the first movable object moves in the trajectory, the dynamic image will move on the map, and the position of the first movable object in the target space will be displayed in real time on the map. The dynamic icon can be any movable image that can identify the first movable object. For example, the dynamic icon can be an avatar registered online by the first movable object, or an icon with the name or ID of the first movable object.

[0080] Regardless of which optional method is used, an optional implementation method of associating the trajectory of the first movable object with the detailed behavior data of the first movable object in the target space includes: determining the behavior object, behavior position and / or behavior time of the first movable object performing the first behavior based on the detailed behavior data of the first movable object in the target space; establishing a corresponding relationship between the behavior object, behavior position and / or behavior time of the first behavior and the trajectory segment of the first movable object where the first behavior is located. In each embodiment of the present application, "and / or" represents at least one of the objects connected by "and / or".

[0081] In the embodiment of the present application, the detailed behavior data of the first movable object in the target space refers to the detailed data of the first behavior of the first movable object in the target space, which can reflect or reflect information such as the behavior object, behavior location and / or behavior time involved in the first behavior of the first movable object in the target space. Depending on the application scenario, target space and first movable object, the first behavior will also have different definitions. The following examples are provided:

[0082] For example, taking the target space as an offline store and the first movable object as a user who enters the store to shop, the first behavior mainly refers to the user's behavior of purchasing items in the store; accordingly, the behavior details data of the first movable object in the target space is the order information formed by the user purchasing items in the offline store. The order information generally includes: the name, price, quantity, payment checkout time, POS machine information for payment checkout, address and name of the offline store, and other information. Among them, the name of the item can directly reflect the item purchased by the user (i.e., the behavior object), and can indirectly reflect the location where the user purchased the item (i.e., the behavior location). The location of the item in the store is known. The payment checkout time can reflect the time when the user purchased the item (i.e., the behavior time) to a certain extent.

[0083] For another example, if the target space is a public place such as a train station or airport, and the first movable object is a passenger entering these public places, then the first behavior can be the passenger passing through the gate to prepare to board a train or plane, or the passenger checking in luggage, or the passenger checking in tickets, etc., which can be flexibly defined according to application requirements. Accordingly, the detailed data of the first movable object's behavior in the target space is the passenger's ticket purchase information. This ticket purchase information generally reflects information such as the train or flight number, departure or take-off time, waiting room or boarding gate, etc. This information can, to a certain extent, reflect the passenger's information about passing through the gate (gate location, gate time, etc.), the passenger's information about checking in luggage (such as window location, approximate time range), or the passenger's ticket check time and location.

[0084] Optionally, the behavior object and / or behavior position can be highlighted in the trajectory segment where the first behavior is located; the behavior time can be marked on the timeline of the trajectory. There is no limitation on the way to highlight the behavior object and / or behavior position. For example, the behavior object and / or behavior position can be framed in the trajectory; or an indicator icon (such as an arrow or a small hand) can be added to the trajectory to point to the behavior object and / or behavior position; or some image or video processing technology can be used to highlight the behavior object and / or behavior position; and so on. There is no limitation on the way to mark the behavior time on the timeline. For example, the behavior time where the first behavior occurs can be framed on the timeline; or the timeline position corresponding to the behavior time can be bolded, highlighted, or the color can be changed; or an indicator icon (such as an arrow or a small hand) can be added to point to the timeline position corresponding to the behavior time; and so on.

[0085] In the embodiment of the present application, the display terminal 13 is required to perform at least: an information acquisition operation and an associated display operation. The information acquisition operation is an abbreviation for "acquiring the target trajectory and at least one associated data object from the server device 12 in response to a query operation"; the associated display operation is an abbreviation for "displaying at least one data object in the process of displaying the target trajectory." In the embodiment of the present application, the detailed implementation methods of the "information acquisition operation" and the "associated display operation" are not limited; any implementation method that can achieve the above-mentioned purpose is applicable to the embodiment of the present application. The detailed implementation methods of the "information acquisition operation" and the "associated display operation" are exemplified below.

[0086] Exemplary description of the implementation method of "information acquisition operation":

[0087] In optional method d1, the display terminal 13 has an electronic screen, which can display a first query interface to the video processing personnel. The first query interface is an interactive interface between the video processing personnel and the display terminal 13. Through the first query interface, the video processing personnel can initiate a first query operation to the display terminal 13.

[0088] Optionally, the first query interface includes several information items, which can lock the target movable object or target trajectory. For example, these information items include but are not limited to: information items describing the trajectory, information items describing the movable object, information items describing the behavior details data, or information items describing the time of entering and leaving the target space, etc. Taking the video screening in the offline retail scene as an example, Figure 3a The figure shows a schematic diagram of the first query interface. Figure 3a In the first query interface, the information items include: time of entering and leaving the store, quantity of ordered goods, etc. These information items can lock the target track that needs to be checked. Figure 3b As shown, the right side of these information items includes a "query" control, and the investigator can click the "query" control to issue the first query operation. Figure 3a In the interface shown, these information items are obtained by querying the order through the order query interface. Figure 3b Shown is another implementation of the first query interface. Figure 3b In the first query interface, information items such as entry and exit time, order number, and user image are included. After the user enters the information required for one or more information items, he or she clicks the "Query" control on the interface to issue the first query operation.

[0089] In response to the first query operation, the display terminal 13 may send a first data acquisition request to the server device 12, requesting the target trajectory. The first data request carries first identification information pointing to the target trajectory, allowing the server device 12 to determine which trajectory to return. The server device 12 then receives the target trajectory and at least one data object associated with the target trajectory returned by the server device 12. The first identification information may be information from the information items carried in the first query interface, such as an identifier of the target movable object, an identifier of detailed behavioral data of the target movable object, or time data of entry and exit of the target space.

[0090] In optional mode d2, the display terminal 13 has an audio component that supports voice recognition. Based on this, the video processing personnel can call the audio component of the display terminal 13 (such as a microphone) to initiate a first query operation to the display terminal 13 in voice. In response to the first query operation, the display terminal 13 can send a first data acquisition request to the server device 12 to request the target trajectory. The first data request carries first identification information pointing to the target trajectory, so that the server device 12 can determine which trajectory needs to be returned; then, the target trajectory returned by the server device 12 and at least one data object associated with the target trajectory can be received. The first identification information can be provided by the video processing personnel in voice, for example, it can be an identification of the target movable object, an identification of the behavior details data of the target movable object, or time data of entering and leaving the target space, etc.

[0091] Exemplary description of the implementation of "association display operation":

[0092] Depending on the application scenario, target space, and movable object, the behavior of the movable object within the target space will vary. Accordingly, the at least one data object that can reflect the behavior of the movable object within the target space will also vary. Depending on the data object, the implementation method for the associated display of the at least one data object during the target trajectory display process will also vary. In the following embodiments, the implementation method of the associated display operation is exemplified by using the example of at least one data object including: time data of the target movable object entering and exiting the target space, detailed data on the target movable object's behavior within the target space, and / or map data of the target space.

[0093] In the optional manner e1, during the display of the target trajectory, the time data of the target movable object entering and exiting the target space are associated and displayed.

[0094] In the optional manner e2, during the display of the target trajectory, detailed behavior data of the target movable object in the target space are associated and displayed.

[0095] In the optional manner e3, during the display of the target trajectory, map data of the target space is associated and displayed.

[0096] In optional manner e4, during the display of the target trajectory, the time data of the target movable object entering and exiting the target space and the behavior detail data of the target movable object in the target space are associated and displayed.

[0097] In the optional mode e5, during the display of the target trajectory, the time data of the target movable object entering and exiting the target space and the map data of the target space are associated and displayed.

[0098] In the optional manner e6, during the display of the target trajectory, the behavior detail data of the target movable object in the target space and the map data of the target space are associated and displayed.

[0099] In optional mode e7, during the display of the target trajectory, the time data of the target movable object entering and exiting the target space, the behavior details data of the target movable object in the target space, and the map data of the target space are associated and displayed.

[0100] In the embodiment of the present application, the display interface style of the display terminal 13 is not limited, and any interface style that can display at least one data object in association with the target trajectory is applicable to the embodiment of the present application. Figure 4a-4d The figure shows a display interface style, which includes: a track display area and an information display area; the track display area is used to display the target track; the information display area is used to display part or all of the data objects associated with the target track. Figure 4a-4d In the article, a display interface style is given by taking the video screening of offline retail scenarios as an example. This interface style is only an example and is not limited to this.

[0101] Regardless of which optional method is used, in the target trajectory display process, an optional implementation method of associating and displaying the time data of the target movable object entering and leaving the target space includes: displaying the time data of the target movable object entering and leaving the target space on the time axis of the target trajectory. Figure 4a-4d As shown, the start time of the timeline is the entry time, and the end time of the timeline is the exit time.

[0102] Regardless of which optional method is used, an optional implementation method of associating and displaying the map data of the target space during the target trajectory display process includes: displaying the map data of the target space during the target trajectory display process; and displaying a dynamic icon in the map data, the dynamic icon being linked to the target movable object in the target trajectory. Figure 4a-4d As shown, the dynamic icon is the user's avatar in the target trajectory, and the position of the avatar on the map represents the user's location in the offline store.

[0103] Furthermore, the map data of the target space can be displayed in the information display area outside the trajectory display area, such as Figure 4a-4c Alternatively, the map data of the target space can be displayed in a floating layer above the trajectory display area, as shown in Figure 4d shown.

[0104] Furthermore, detailed behavior data of the target movable object within the target space refers to detailed data regarding the target movable object's first behavior within the target space, and may reflect or embody information such as the target object, location, and / or time of the first behavior. The definition of "first behavior" may vary depending on the application scenario, target space, and target movable object, and is not limited thereto.

[0105] When the behavior details data of the target movable object in the target space includes: the behavior object, behavior location, and / or behavior time of the first behavior of the target movable object in the target space, during the target trajectory display process, the associated display of the behavior details data of the target movable object in the target space includes at least one of the following operations:

[0106] Operation A1: On the time axis of the target trajectory, display the behavior time when the target movable object performs the first behavior in the target space.

[0107] Operation A2: Displaying, in the information display area outside the target trajectory display area, a behavior object of the target movable object performing the first behavior in the target space;

[0108] Operation A3: Marking a behavior position where the target movable object performs a first behavior in the target space in the map data.

[0109] Regarding operation A1: Mark the time when the first behavior occurs on the time axis of the target track, that is, the position on the time axis corresponding to the behavior time will be marked. Figure 4a-4d As shown, the location of the icon "*" on the timeline indicates the time when the first behavior occurred. Furthermore, the marked locations on the timeline are optionally interactive. Relevant processing personnel can initiate a trigger operation on the marked location on the timeline to quickly locate the trajectory segment when the target movable object first behaved. Trigger operations include, but are not limited to, clicking, double-clicking, touching, hovering over the mouse, or long pressing.

[0110] Regarding operation A2: in the information display area outside the trajectory display area, the behavior object of the target movable object performing the first behavior in the target space is displayed. Figure 4a-4d As shown, the items in the order details on the right are the behavior objects involved when the first behavior is a shopping behavior. Optionally, the behavior objects involved in the first behavior can be displayed in the form of an object list. Furthermore, the behavior objects are interactive, and relevant processing personnel can initiate a trigger operation on the behavior objects displayed in the information display area to quickly locate the trajectory segment where the behavior object is located. The trigger operations here include but are not limited to: click, double-click, touch, mouse hover or long press, etc.

[0111] Regarding operation A3: As the target track is displayed, when a track segment containing a behavior object is displayed, the behavior position where the first behavior occurs in the track segment is marked in the map data. Figure 4a-4d As shown, the position of the user's avatar in the map is the position of the user when the item is selected, that is, the position when the purchase occurs. The position of the user's avatar in the map will change dynamically as the target track is played.

[0112] Furthermore, the process of associating and displaying at least one data object may further include at least one of the following operations:

[0113] Operation B1: When a track segment containing a behavior object is displayed, highlight the behavior object in the information display area. Figure 4b As shown, when the video clip of the user selecting apples (i.e., the track clip) is played, the apple information in the right information display area is highlighted. In operation B1, the highlighting method is not limited and can be a frame selection method, a highlighting method, or an animation method.

[0114] Operation B2: When displaying a track segment containing a behavior object, highlight the behavior time when the first behavior occurs for the behavior object on the timeline. Figure 4b As shown, when the video clip (i.e., the trajectory clip) of the user selecting apples is played, the corresponding time position on the timeline is indicated by a small hand icon. In operation B2, the highlighting method is not limited and can be used to highlight the time position displayed on the timeline by framing, highlighting, or indicating with an icon (e.g., an arrow or a small hand icon).

[0115] Operation B3: In response to the triggering operation of the behavior object in the information display area, jump from the current display position to the trajectory segment containing the behavior object. Figure 4c As shown, in response to a click operation on the tomato information displayed in the information display area on the right, the video clip of the user purchasing apples (i.e., the trajectory clip) can be jumped from the currently playing video clip of the user purchasing tomatoes (i.e., the trajectory clip). Figure 4c Shown is a video clip (i.e., trajectory clip) of a user selecting tomatoes.

[0116] Operation B4: In response to a triggering operation on any behavior time on the timeline, jump from the current display position to the track segment corresponding to the triggered behavior time.

[0117] In an optional embodiment, the map data also includes icons of multiple cameras in the target space, such as Figure 4a-4dBased on this, during the display of the target trajectory, the movement of the target movable object can also be followed, and the camera that captured the target movable object can be dynamically marked in the map data. Figure 4c and Figure 4d The camera encircled by the dotted circle icon is the camera that captured the target movable object in the current video clip.

[0118] In order to facilitate a clearer understanding of the technical solutions provided by the embodiments of the present application, in the following embodiments, the technical solutions of the present application are described in detail and completely by taking the video screening of offline retail scenarios as an example.

[0119] like Figure 2 As shown, in an offline retail scenario, the target space is an offline store 20, in which multiple cameras 21 are deployed. The offline store 20 contains items 24 for sale, which are placed on shelves 23. Aisles are formed between the shelves 23, and users can move in the aisles to browse, select, and purchase items 24 on the shelves 23. Figure 2 As shown, a POS machine 27 is provided near the exit of the offline store 20 to provide self-service payment services to users.

[0120] Multiple cameras 21 capture video data in offline stores, and the video data is uploaded to the server device 25 corresponding to the offline store 20, such as Figure 2 As shown. The server device 25 can be a conventional server, a server array or a cloud server. Figure 2 In the embodiment, the server device 25 is illustrated by taking a cloud server as an example, but the present invention is not limited thereto. The server device 25 stores video data captured by multiple cameras 21.

[0121] In offline retail scenarios, self-service payment often results in missed or unpaid payments, which can lead to asset losses. This can be addressed by reviewing videos captured by multiple cameras 21 to identify losses. However, the video data captured by multiple cameras 21 is large in volume and duration, making reviewing all video data both costly and inefficient.

[0122] In this embodiment, the server device 25, on the one hand, merges and processes the video data captured by multiple cameras 21, taking each user entering the offline store 20 as a unit, to obtain a trajectory video of each user. On the other hand, it also associates each user's trajectory video with the time the user entered and exited the offline store 20, the order data generated by the user in the offline store 20, and the map of the offline store 20, providing a data foundation for subsequent video investigation. With this data foundation, when video investigation is required, investigators can filter out suspicious orders, suspicious users, or suspicious time periods. Based on this information, they can filter out the trajectory videos to be investigated, narrowing the scope of the video investigation, which is conducive to improving efficiency and saving costs.

[0123] Optionally, the investigator can Figure 3a or Figure 3b The first query interface shown in FIG. 1 sets the conditions that the trajectory video to be checked needs to meet and reports it to the server device 25. Figure 3a As an example, in the order query interface, you can quickly search and locate suspicious users or suspicious orders by face / body photos, order numbers, account numbers, and other screening conditions. Figure 3a The first query interface shown in the figure displays relevant information items of the trajectory video to be checked, such as face / full body image, store entry time, store exit time, and order information, etc. These information items can uniquely identify a trajectory video. Click the "Query" control next to the corresponding trajectory video to initiate a query request to the server device 25. The server device 25 will filter out the trajectory videos of suspicious users or suspicious orders based on the above information items, and send the trajectory videos of suspicious users or suspicious orders, the time when the suspicious users enter and exit the offline store 20, the order data of the suspicious users, and the location of the offline store 20 to the server device 25. Figure 1 And return it to the terminal device 26 of the investigator.

[0124] The terminal device 26 receives the trajectory video of the suspicious user or suspicious order, the time when the suspicious user enters and leaves the offline store 20, the order data of the suspicious user, and the map of the offline store 20 returned by the server device 25; then, Figure 4a-4d As shown, the trajectory video of the suspicious user or suspicious order is played in the trajectory display area, that is, the video footage of the suspicious user or the user corresponding to the suspicious order in the offline store 20, so that the investigation personnel can check whether the suspicious user in the trajectory video has missed payment, escaped the bill, etc.

[0125] In this embodiment, it is assumed that the suspicious user has made at least one purchase in the offline store 20, and each purchase is referred to as a purchase. Figure 4a-4dAs shown, the time when the suspicious user enters and exits the offline store 20 is marked on the time axis of the trajectory video, and the time point when the suspicious user picks up the goods in the offline store 20 (referred to as the pickup point for short) is marked. Figure 4a-4d The locations marked with an "*" on the timeline are pickup points. For investigators, clicking, touching, or double-clicking these pickup points on the timeline quickly directs the video to the corresponding video clip. Marking the pickup items in the video clip allows investigators to quickly and effectively identify specific pickup activities, improving investigation efficiency. Similarly, when playing a video clip of a pickup, the pickup point can be highlighted.

[0126] Further, if Figure 4a-4d As shown, the information display area on the right can also display the suspicious user's order information within offline store 20, primarily information about all products purchased by the suspicious user within offline store 20. For investigators, by clicking, touching, or long-pressing the corresponding product information, the video can be quickly positioned to the video clip of the suspicious user picking up the product. The product can also be marked in the video clip, allowing investigators to quickly and specifically identify specific product picking behaviors, thereby improving investigation efficiency. Accordingly, when a video clip of a specific product is played, the product information on the right can also be highlighted.

[0127] Further, if Figure 4a-4c As shown, a map of the offline store 20 can also be displayed in the information display area on the right. When a video clip of a certain pickup is played, the pickup location corresponding to the suspicious user can be displayed on the map, and the corresponding pickup point on the timeline can also be highlighted. For investigators, they can also trigger a certain location or area on the map to quickly locate the video screen to the video clip in the corresponding location or area. Similarly, they can also trigger a certain camera on the map to quickly locate the video screen to the video clip captured by that camera. This allows investigators to quickly and specifically view video clips captured by certain locations, areas, or cameras, thereby improving investigation efficiency.

[0128] Further, if Figure 4d As shown, the map of the offline store 20 can be expanded and displayed as a semi-transparent floating layer above the video screen. When the map is expanded, the information on the map is clearer and easier to view and operate. Of course, when the map does not need to be expanded, you can also use certain trigger operations, such as clicking the close control or shrink control in the upper right corner to shrink the map to the information display area on the right.

[0129] In this embodiment, while playing the trajectory video of the suspicious user in the offline store 20, the picking-up point is displayed on the timeline, the picking-up location corresponding to the picking-up point is highlighted on the map of the offline store 20, and the product information corresponding to the picking-up point is highlighted in the picking-up list. This can comprehensively and three-dimensionally display the three-dimensional information of the picking-up, that is, the time, place and product of the picking-up, so as to facilitate the quick and accurate investigation of inventory losses.

[0130] Video completion:

[0131] In the above-mentioned embodiments of the present application, by merging and processing the video data captured by multiple cameras into trajectories based on movable objects, the video processing operation can be greatly simplified, the video processing efficiency can be improved, and the cost can be reduced. However, due to technical reasons such as the on-site environment in the target space and the video merging process, there may be missing trajectories. In order to ensure the integrity of the trajectory and the effect of tracing back or troubleshooting based on the trajectory, in some embodiments of the present application, before displaying the target trajectory, it can also be determined whether there are missing parts in the target trajectory; if there are missing parts, the target trajectory is completed. The completion here refers to the technology of completing the missing parts in the target trajectory.

[0132] In an embodiment of the present application, a video completion solution is provided to complete a target trajectory. Specifically, the display terminal 13 can display the target trajectory with missing portions. Furthermore, based on the direction of the target movable object before the missing portion, candidate cameras within the target space that are likely to have captured the missing portion are identified. The target trajectory is then completed based on the video content captured by the candidate cameras that corresponds to the missing time period. The missing time period is the time period corresponding to the missing portion.

[0133] Furthermore, based on the above-mentioned video completion solution, the embodiment of the present application also provides a human-computer interactive video completion solution to complete the target trajectory. Specifically, for the display terminal 13, on the one hand, it can display the target trajectory with missing parts; on the other hand, it can respond to the completion operation initiated by the user for the missing part, and determine the candidate cameras in the target space that may have captured the missing part according to the direction of the target movable object before the missing part; play the video content corresponding to the missing time period captured by the candidate camera for the user to check whether it contains the target movable object; and respond to the completion confirmation operation initiated by the user, use the trajectory segment containing the target movable object to complete the target trajectory. Among them, the user who initiates the completion operation on the target trajectory can be a video processing personnel, or other personnel specifically responsible for video completion operations, and there is no limitation on this.

[0134] In the embodiments of the present application, the method by which a user initiates a completion operation is not limited. For example, a user can initiate a completion operation by issuing a completion command to the display terminal via voice. For another example, a completion control can be provided on the interface displaying the target trajectory, and the user can trigger the completion control to initiate a completion operation on the target trajectory. For another example, before displaying the target trajectory, the missing portion of the target trajectory and its corresponding missing time period can be determined based on the time when the target movable object enters and exits the target space and the time when the target movable object appears in the target trajectory. During the display of the target trajectory, the missing time period is marked on the timeline of the target trajectory. The missing time period on the timeline has an interactive function, and the user can trigger the missing time period on the timeline to initiate a completion operation. The display terminal 13 can respond to the user's triggering operation on the missing time period on the timeline and determine that the user has initiated a completion operation for the missing portion of the target trajectory. The triggering operation described in this description includes, but is not limited to, clicking, double-clicking, touching, hovering over the mouse, or long pressing.

[0135] In some optional embodiments, in the process of displaying the target trajectory, a map of the target space can be associated; the map displays a camera in the target space and a dynamic icon, which is linked to the target movable object in the target trajectory. In these embodiments, the map can be combined to determine the candidate camera from the cameras displayed on the map. An implementation method for determining the candidate camera includes: responding to a completion operation initiated by the user for a missing part, calculating the last direction of the target movable object before the missing part; adjusting the orientation of the camera on the map that last photographed the target movable object before the missing part to be consistent with the last direction; responding to a selection operation initiated by the user for a camera within the coverage range of the orientation, determining the camera selected by the user as a candidate camera. The orientation of the camera that last photographed the target movable object before the missing part is as follows: Figure 5 shown.

[0136] Furthermore, the candidate camera is most likely a camera that has not captured the target movable object. Based on this, before responding to the user's selection operation initiated for the cameras within the coverage area, the display terminal can also mark the cameras that have not captured the target movable object on the map, so that the user can initiate a selection operation for the cameras within the coverage area that have not captured the target movable object. Further optionally, the display terminal can also mark the cameras that have captured the target movable object, or mark the cameras that have captured the target movable object and the cameras that have not captured the target movable object separately. Regardless of the marking method, it can assist the user in identifying the cameras that have not captured the target movable object, making it easier for the user to select candidate cameras from them.

[0137] In the embodiments of the present application, the method of completing the target trajectory based on the video content corresponding to the missing time period captured by the candidate camera is not limited. For example, the display terminal can directly use the characteristics of the target movable object to adapt the video data captured by the candidate camera; if a trajectory segment containing the target movable object is found and the shooting time of the trajectory segment corresponds to the missing time period, the target trajectory can be completed using the trajectory segment; conversely, if no trajectory segment containing the target movable object is found or the shooting time of the trajectory segment containing the target movable object does not correspond to the missing time period, the completion operation can be terminated, or a prompt message can be output for the user to reselect a candidate camera for the next completion operation.

[0138] In addition to the above embodiment, in another optional embodiment, the display terminal may display the video content captured by the candidate camera corresponding to the time period to be played, allowing the user to check whether the video content contains the target movable object. The time period to be displayed at least includes the missing time period. If the user finds that the target movable object is included, a completion confirmation operation can be initiated; otherwise, no completion confirmation operation is issued and a new candidate camera can be selected. If the user initiates the completion confirmation operation, the display terminal can respond to the user-initiated completion confirmation operation and complete the target trajectory using the trajectory segment containing the target movable object.

[0139] Furthermore, when playing the video content captured by the candidate camera corresponding to the time period to be displayed, the specified video content captured by the candidate camera can be played first, and the missing time period can be marked on the timeline of the specified video content. In response to the user's operation of selecting a time point forward or backward from the missing time period, the start and end time points of the time period to be displayed are determined. Then, starting from the start time point, the video content captured by the candidate camera is played until the end time point. Optionally, the specified video content can be all the video content captured by the candidate camera, or video content captured on the day in which the missing time period occurs.

[0140] In the embodiment of the present application, by completing the trajectory, the trajectory of the target movable object in the target space can be fully and completely displayed, which can provide a complete data basis for subsequent trajectory-based investigation or other data analysis, and is conducive to improving the accuracy of trajectory-based investigation or other data analysis.

[0141] It should be noted that the above-mentioned video completion process can also be implemented independently and does not necessarily rely on the content of the previous embodiment. In other words, any application scenario that requires completing the trajectory of movable objects can adopt the video completion solution provided by the embodiment of this application or its variant, without any restrictions on how to obtain the trajectory of movable objects or whether to further associate and display at least one data object after video completion.

[0142] Video processing for the second behavior:

[0143] In some embodiments of the present application, after obtaining the trajectory of each movable object, the server device 12 may further analyze the trajectory of each movable object to determine whether each movable object exhibits or may exhibit a second behavior. Furthermore, if a movable object exhibits or may exhibit a second behavior, it may add tag information related to the second behavior to its trajectory, thereby facilitating subsequent video processing based on the second behavior. The operation of associating the trajectory of a movable object with at least one data object and the operation of adding tag information related to the second behavior to the trajectory of the movable object may be associated with each other or performed independently.

[0144] The second behavior is different from the first behavior and can vary depending on the application scenario, target space, and movable objects. For example, in an offline retail scenario, the first behavior is a user purchasing goods in an offline store. Accordingly, the second behavior includes, but is not limited to, at least one of: missed payment, unpaid, overpaid, small-amount ordering, and purchasing designated items. Missed payment refers to a user picking up multiple items in an offline store and only paying for some (but not all) of them. Unpaid refers to a user picking up items in an offline store but not paying for them. Small-amount ordering refers to a user purchasing items in an offline store and paying less than a set threshold, such as a payment of less than 0.4 yuan, or initiating payment only for specific small-amount items (such as shopping bags). Purchasing designated items refers to a user purchasing designated items in an offline store. The designated items can be flexibly set based on application requirements, such as fresh produce like fish, meat, and fruit, or grains and oils like rice and flour. Overpaid refers to a user purchasing items in an offline store and paying more than the actual amount of the items purchased.

[0145] For different movable objects, the server device 12 uses the same or similar process to analyze whether the second behavior exists or may exist based on the object's trajectory. In the following embodiments, the first movable object is used as an example. The first movable object can be any movable object that enters or exits the target space. That is, after obtaining the trajectory of the first movable object, the server device 12 can further analyze whether the first movable object exists or may exist the second behavior based on the trajectory of the first movable object; if so, it adds tag information related to the second behavior to the trajectory of the first movable object.

[0146] Optionally, the server device 12 may use a pre-trained recognition model and input the trajectory of the first movable object into the recognition model to obtain a recognition result of whether the first movable object has or may have a second behavior. Alternatively, some behavioral features related to the second behavior may be provided to the server device 12 in advance; the server device 12 may compare and analyze the trajectory of the first movable object with these features to obtain a recognition result of whether the first movable object has or may have a second behavior. Alternatively, some features or identifiers (such as names, etc.) of movable objects that may have a second behavior may be provided to the server device 12 in advance; the server device 12 may compare and analyze these features or identifiers with the first movable object in the trajectory to obtain a recognition result of whether the first movable object has or may have a second behavior.

[0147] Further optionally, the second behavior also involves information such as the behavior object, behavior time, and behavior location. Based on this, adding marking information related to the second behavior to the trajectory of the first movable object includes at least one of the following operations:

[0148] Operation C1: adding a highlight mark to the behavior object of the second behavior in the trajectory of the first movable object;

[0149] Operation C2: adding at least one of detailed information of the behavior object of the second behavior, online sales information, offline sales information, offline inventory, and replenishment suggestion information to the trajectory of the first movable object;

[0150] Operation C3: adding a highlight mark for the behavior time of the second behavior in the trajectory of the first movable object;

[0151] Operation C4: adding a highlight mark to the behavior position of the second behavior in the trajectory of the first movable object;

[0152] Operation C5: adding a highlight mark to the first movable object in the trajectory of the first movable object;

[0153] Operation C6: Add detailed information of the first movable object to the trajectory of the first movable object.

[0154] by Figure 2 Take the offline retail scenario shown as an example, Figure 6a As shown, the user in the trajectory is the first movable object, and detailed information of the user can be added to the trajectory, such as the user's store entry time, store exit time, the user's online account, the user's order number, the user's avatar, etc. Figure 6a As shown, detailed information of the goods picked up and other information can also be added to the user's trajectory. The goods picked up are the behavior objects involved in the second behavior. The detailed information of the goods picked up includes but is not limited to: product name, product number, product status, etc. Other information of the goods includes but is not limited to: online sales information (for example, 1000 pieces), offline sales information (for example, 100 pieces), offline inventory (for example, 100 pieces) and replenishment suggestion information (for example, replenish 100 pieces). Further, as Figure 6a As shown, on the time axis of the trajectory, the highly suspicious picking-up points are marked with the icon "solid circle", and the highly suspicious picking-up points are the time when the second behavior (such as picking up designated items or picking up missed items) occurs.

[0155] Based on the above-mentioned trajectory with the marking information related to the second behavior, the video processing personnel can process the trajectory segment of the movable object with the second behavior. For the display terminal 13, according to the second query operation, a second data request can be sent to the server device 12 to request the target trajectory segment marked with the target behavior; the second data request includes second identification information pointing to the target behavior, and the target behavior belongs to the second behavior. For the server device 12, the second data request sent by the display terminal 13 can be received; according to the second identification information, the target trajectory segment containing the target behavior is obtained from the trajectory of at least one movable object; the target trajectory segment and the marking information related to the target behavior in the target trajectory segment are provided to the display terminal 13. The display terminal 13 receives the target trajectory segment returned by the server device 12 and the marking information related to the target behavior in the target trajectory segment; during the display process of the target trajectory segment, the marking information related to the target behavior in the target trajectory segment is associated and displayed.

[0156] It should be noted that there may be one or more target trajectory segments containing the target behavior. In the case of multiple target trajectory segments, these target trajectory segments may come from the trajectories of different movable objects.

[0157] It should be noted that the video processing personnel can process the trajectory fragment of the movable object with the second behavior separately, or can process the trajectory of the movable object at the same time. In an optional embodiment, the video processing personnel can first process the target trajectory fragment containing the target behavior, and further determine whether it is necessary to view the complete trajectory and process the complete trajectory based on the processing result of the target trajectory fragment. Based on this, an implementation method of sending a first data request to the server device in response to the first query operation includes: during the display of the target trajectory fragment, displaying a control for viewing the complete trajectory; in response to the triggering operation of the control, sending a first data request to the server device; wherein the complete trajectory to which the target trajectory fragment belongs is the target trajectory in the aforementioned embodiment.

[0158] In the embodiment of the present application, the embodiment of the display terminal 13 sending the second data request to the server device 12 is not limited. The following example illustrates:

[0159] In optional method f1, the display terminal has an electronic screen, which can display a second query interface to the video processing personnel. The second query interface is an interactive interface between the video processing personnel and the display terminal. Through the second query interface, the video processing personnel can initiate a second query operation to the display terminal.

[0160] Optionally, the query interface includes several query condition information items, which can lock the target trajectory segment. For example, it can be the identification information item of the trajectory, the identification information item of the movable object, the identification information item of the behavior details data, or the time information item of entering and leaving the target space, etc. Figure 6b As shown in FIG, a schematic diagram of the second query interface is given by taking the video screening in the offline retail scene as an example. Figure 6b The second query interface shown includes information items such as face / full body image, product number, shelf number, and camera.

[0161] exist Figure 6b In the process, the investigator can manually enter the corresponding information in the information item on the second query interface, and then click the query control on the second query interface to issue a second query operation to the display terminal 13.

[0162] Further, if Figure 6bAs shown, in offline retail scenarios, the second query interface includes not only query condition information items but also map data of offline stores. Based on this, investigators can directly trigger shelf information, store locations, and / or cameras in the map data. The display terminal can respond to the triggering operation of shelf information, store locations, and / or cameras in the map data, filling the triggered shelf information, store locations, and / or camera information as query conditions; then, based on the query conditions, a second data request is sent to the server device, so that the server device can return the target trajectory segment marked with the target behavior based on the query conditions.

[0163] In optional approach f2, display terminal 13 includes an audio component that supports voice recognition. Based on this, the video processing personnel can use the audio component of display terminal 13 (e.g., a microphone) to initiate a second query operation to the display terminal via voice. In response to the second query operation, display terminal 13 can send a second data acquisition request to server device 12, requesting target trajectory segments marked with the target behavior.

[0164] In the embodiment of the present application, the display terminal 13 is not limited to the implementation method of the associated display of the target behavior in the target trajectory segment. For example, the target movable object, the behavior object related to the second behavior, the behavior position and the behavior time can be highlighted during the display of the target trajectory segment. Figure 6a shown.

[0165] In addition to the above-mentioned video processing system, for various application scenarios where multiple cameras are deployed, the embodiments of the present application also provide a video processing method, a video display method, and a video completion method, etc. These methods are described in detail below through different embodiments.

[0166] Figure 7a The flowchart of a video processing method provided by the exemplary embodiment of the present application is as follows. Figure 7a As shown, the method includes:

[0167] 71a. Obtain video data captured by multiple cameras in a target space, where the target space contains at least one movable object.

[0168] 72a. Taking movable objects as units, merge the video data captured by multiple cameras to obtain a trajectory of at least one movable object.

[0169] 73a. Associate the trajectory of at least one movable object with at least one data object, where the at least one data object is related to the behavior of the movable object in the target space.

[0170] In this embodiment, the target space refers to a physical space that accommodates and allows movable objects to enter and exit. A movable object broadly refers to any object that can move, including autonomous objects such as users, robots, or unmanned vehicles, as well as non-autonomous objects such as human-driven vehicles. The specific implementation of the target space will vary depending on the application scenario, and accordingly, the movable objects that can enter and exit the target space will also vary.

[0171] In some application scenarios, the target space is an offline store such as a shopping mall or supermarket. Correspondingly, the movable objects are users entering and exiting the offline store. The users here include consumers and / or supermarket staff, or may also include autonomous shopping carts that can move freely in the offline store, or robots, etc.

[0172] Multiple cameras are installed in the target space. The multiple cameras are installed at different positions in the target space, have different fields of view, and can capture video data within their respective fields of view. The multiple cameras can upload the captured video data to the execution subject of the method of this embodiment, such as a server device. The server device merges the video data captured by the multiple cameras in units of movable objects to obtain the trajectory of at least one movable object, and then, the trajectory of at least one movable object can be associated with at least one data object, respectively, providing conditions for video processing (such as troubleshooting, editing or beautification) in units of movable objects, which is conducive to simplifying video processing (such as troubleshooting, editing or beautification) operations.

[0173] In an optional embodiment, the trajectory of at least one movable object is associated with at least one data object, including: for a first movable object, associating the trajectory of the first movable object with time data of the first movable object entering and exiting a target space, behavior detail data of the first movable object in the target space and / or map data of the target space; wherein the first movable object is any movable object among the at least one movable object.

[0174] Optionally, associating the trajectory of the first movable object with the time data of the first movable object entering and exiting the target space includes: generating time axis information of the trajectory of the first movable object according to the time data of the first movable object entering and exiting the target space.

[0175] Optionally, associating the trajectory of the first movable object with the map data of the target space includes adding a dynamic icon to the map data of the target space based on the trajectory of the first movable object, wherein the dynamic icon is linked to the first movable object. For example, the dynamic icon may be an avatar registered online by the first movable object.

[0176] Optionally, the above-mentioned associating the trajectory of the first movable object with the behavior details data of the first movable object in the target space includes: determining the behavior object, behavior position and / or behavior time of the first movable object performing the first behavior based on the behavior details data of the first movable object in the target space; establishing a correspondence between the behavior object, behavior position and / or behavior time of the first behavior and the trajectory segment where the first behavior is located in the trajectory of the first movable object.

[0177] In an optional embodiment, the method of this embodiment further includes: analyzing whether the first movable object has or may have a second behavior based on the trajectory of the first movable object; if so, adding marking information related to the second behavior to the trajectory of the first movable object.

[0178] Optionally, adding marking information related to the second behavior to the trajectory of the first movable object includes at least one of the following operations:

[0179] Adding a highlight mark to the behavior object of the second behavior in the trajectory of the first movable object;

[0180] Adding at least one of detailed information of the behavior object of the second behavior, online sales information, offline sales information, offline inventory, and replenishment suggestion information to the trajectory of the first movable object;

[0181] Adding a highlight mark for the behavior time of the second behavior in the trajectory of the first movable object;

[0182] Adding a highlight mark for the behavior position of the second behavior in the trajectory of the first movable object;

[0183] In the trajectory of the first movable object, adding a highlight mark for the first movable object;

[0184] In the track of the first movable object, detailed information of the first movable object is added.

[0185] In an optional embodiment, the target space is an offline store, which contains items for sale; the behavior detail data is the order information formed by the first movable object selecting items in the offline store; the first behavior is the behavior of selecting items; accordingly, the second behavior includes: at least one of: missed payment behavior, unpaid behavior, overpayment behavior, small order behavior and behavior of purchasing designated items.

[0186] In an optional embodiment, the method of this embodiment further includes: receiving a second data request sent by the display terminal, the second data request including second identification information pointing to the target behavior, the target behavior belonging to the second behavior; obtaining a target trajectory segment containing the target behavior from the trajectory of at least one movable object based on the second identification information; and providing the target trajectory segment and the marking information related to the target behavior in the target trajectory segment to the display terminal, so that the display terminal can associate and display the target trajectory segment and the marking information related to the target behavior contained therein.

[0187] In an optional embodiment, the method of this embodiment further includes: receiving a first data request sent by a display terminal, the first data request including first identification information pointing to a target trajectory; obtaining the target trajectory from the trajectory of at least one movable object based on the first identification information; and providing the target trajectory and at least one data object associated therewith to the display terminal, so that the display terminal can associate and display the target trajectory and the at least one data object associated therewith.

[0188] Optionally, the first identification information is an identification of the target movable object, or an identification of behavior detail data of the target movable object; the target movable object is a movable object corresponding to the target track.

[0189] In an optional embodiment, the above-mentioned merging and processing of the video data captured by multiple cameras in units of movable objects to obtain the trajectory of at least one movable object includes: performing target tracking on the video data captured by each of the multiple cameras to obtain the movable objects captured by each of the multiple cameras; performing REID processing on the movable objects captured by each of the multiple cameras to obtain the trajectory of at least one movable object.

[0190] Optionally, the above-mentioned target tracking is performed on the video data captured by each of the multiple cameras to obtain the movable objects captured by each of the multiple cameras, including: for each camera, a target tracking operation is performed every N frames of video data, and in each target tracking operation, the N frames of video data are combined to determine whether the movable object tracked by the previous target tracking operation is tracked; wherein N is an integer greater than or equal to 2.

[0191] Optionally, performing REID processing on the movable objects captured by each of the multiple cameras to obtain a trajectory of at least one movable object includes:

[0192] Combined with the overlapping fields of view between multiple cameras, the spatiotemporal information of the movable objects and / or the feature information of the movable objects obtained by other means, REID processing is performed on the movable objects captured by each of the multiple cameras to obtain the trajectory of at least one movable object.

[0193] Further optionally, feature information of the movable object is obtained by other means, including at least one of the following: obtaining a facial image of the movable object captured by a POS machine in the target space; obtaining a facial image in an electronic account corresponding to the movable object.

[0194] Optionally, the movable object in this embodiment may be a user, a robot, or an autonomous shopping cart in the target space, but is not limited thereto.

[0195] In this embodiment, the video data captured by multiple cameras in the target space are merged and processed based on the movable objects entering the target space to obtain the trajectory of each movable object, and the trajectory of each movable object is associated with the data object related to its behavior in the target space, which provides conditions for video processing based on movable objects, is conducive to simplifying video processing operations and improving video processing efficiency.

[0196] Figure 7b FIG. 1 is a flow chart of another video processing method provided by an exemplary embodiment of the present application. Figure 7b As shown, the method includes:

[0197] 71b. Obtain video data captured by multiple cameras in a target space, where the target space contains at least one movable object.

[0198] 72b. Taking movable objects as units, merge the video data captured by multiple cameras to obtain a trajectory of at least one movable object.

[0199] 73b. Analyze, based on the trajectory of each movable object, whether each movable object has or may have a second behavior.

[0200] 74b. Add marking information related to the second behavior to the trajectory of the movable object where the second behavior exists or may exist.

[0201] For the description of steps 71b and 72b, please refer to the aforementioned embodiment and will not be repeated here.

[0202] In this embodiment, the methods for analyzing whether different movable objects have or may have the second behavior are the same or similar. For ease of description, the process of analyzing whether a movable object has or may have the second behavior is described in detail using a first movable object as an example. The first movable object is any one of the at least one movable object.

[0203] Specifically, it is possible to analyze whether the first movable object has or may have the second behavior based on the trajectory of the first movable object; if so, mark information related to the second behavior is added to the trajectory of the first movable object.

[0204] In an optional embodiment, adding marking information related to the second behavior to the trajectory of the first movable object includes at least one of the following operations:

[0205] Adding a highlight mark to the behavior object of the second behavior in the trajectory of the first movable object;

[0206] Adding at least one of detailed information of the behavior object of the second behavior, online sales information, offline sales information, offline inventory, and replenishment suggestion information to the trajectory of the first movable object;

[0207] Adding a highlight mark for the behavior time of the second behavior in the trajectory of the first movable object;

[0208] Adding a highlight mark for the behavior position of the second behavior in the trajectory of the first movable object;

[0209] In the trajectory of the first movable object, adding a highlight mark for the first movable object;

[0210] In the track of the first movable object, detailed information of the first movable object is added.

[0211] In an optional embodiment, the target space is an offline store containing items for sale; the second behavior includes at least one of: missed payment, unpaid, overpaid, a small order, and purchasing a specified item. Accordingly, the movable object may be a user, a robot, or an autonomous shopping cart within the target space, but is not limited thereto.

[0212] In an optional embodiment, the method of this embodiment further includes: receiving a data request sent by a display terminal, the data request including identification information pointing to a target behavior, the target behavior belonging to a behavior; obtaining a target trajectory segment containing the target behavior from the trajectory of at least one movable object based on the identification information; and providing the target trajectory segment and the marking information related to the target behavior in the target trajectory segment to the display terminal, so that the display terminal can associate and display the target trajectory segment and the marking information related to the target behavior contained therein.

[0213] In this embodiment, the video data captured by multiple cameras in the target space are merged and processed based on the movable objects entering the target space to obtain the trajectory of each movable object, and the movable objects that have or may have the second behavior are identified, and marking information related to the second behavior is added to their trajectories. This provides conditions for video processing initiated for the second behavior based on the movable objects, which is conducive to simplifying video processing operations and improving video processing efficiency.

[0214] Figure 8The flowchart of a video display method provided by the exemplary embodiment of the present application is as follows. Figure 8 As shown, the method includes:

[0215] 81. In response to the first query operation, send a first data request to the server device to request a target trajectory; the target trajectory is a trajectory of the target movable object in the target space.

[0216] 82. Receive a target trajectory and at least one data object associated with the target trajectory returned by the server device.

[0217] 83. During the target trajectory display process, at least one data object is associated and displayed, and the at least one data object is related to the behavior of the target movable object in the target space.

[0218] In an optional embodiment, the first data request carries first identification information pointing to the target trajectory, where the first identification information is an identification of a target movable object associated with the target trajectory, an identification of behavior detail data of the target movable object, or time data of entering and exiting the target space.

[0219] In an optional embodiment, the at least one data object includes: time data of the target movable object entering and exiting the target space, detailed behavior data of the target movable object in the target space and / or map data of the target space.

[0220] Optionally, in the target trajectory display process, the associated display of the time data of the target movable object entering and exiting the target space includes: displaying the time data of the target movable object entering and exiting the target space on the time axis of the target trajectory.

[0221] Optionally, the above-mentioned behavior detail data includes: the behavior object, behavior location and / or behavior time of the first behavior of the target movable object in the target space. Based on this, in the target trajectory display process, the associated display of the target movable object's behavior detail data in the target space includes at least one of the following operations:

[0222] On the time axis of the target trajectory, the behavior time when the target movable object takes the first behavior in the target space is displayed;

[0223] Displaying, in an information display area outside the target trajectory display area, a behavior object of the target movable object performing a first behavior in the target space;

[0224] A behavior position where the target movable object performs a first behavior in the target space is marked in the map data.

[0225] Further optionally, marking the behavior position where the target movable object performs the first behavior in the target space in the map data includes: when displaying the trajectory segment containing the behavior object, marking the behavior position where the first behavior occurs in the trajectory segment in the map data.

[0226] Optionally, during the target trajectory display process, the map data of the target space is associated and displayed, including: during the target trajectory display process, the map data of the target space is displayed; and a dynamic icon is displayed in the map data, the dynamic icon being linked to the target movable object in the target trajectory.

[0227] Further optionally, during the target trajectory display process, displaying the map data of the target space includes: displaying the map data of the target space in an information display area outside the target trajectory display area; or displaying the map data of the target space in a floating layer above the target trajectory display area.

[0228] In an optional embodiment, the method of this embodiment further includes at least one of the following operations:

[0229] When displaying a trajectory segment containing a behavior object, highlight the behavior object in the information display area;

[0230] When displaying a trajectory segment containing a behavior object, the behavior time at which the first behavior occurs for the behavior object is highlighted on the timeline;

[0231] In response to a triggering operation on a behavior object in the information display area, jumping from a current display position to a trajectory segment containing the behavior object;

[0232] In response to the triggering operation of any behavior time on the timeline, jump from the current display position to the track segment corresponding to the triggered behavior time.

[0233] In an optional embodiment, the map data includes icons of multiple cameras within the target space. Based on this, the method of this embodiment further includes: during the target trajectory display process, following the movement of the target movable object, dynamically marking the cameras that have captured the target movable object in the map data.

[0234] In an optional embodiment, the method of this embodiment further includes: in response to a second query operation, sending a second data request to the server device to request a target trajectory segment marked with the target behavior; receiving the target trajectory segment returned by the server device and the marking information related to the target behavior in the target trajectory segment; and in the display process of the target trajectory segment, associating and displaying the marking information related to the target behavior in the target trajectory segment.

[0235] Based on the above optional embodiment, one implementation of sending the first data request to the server device in response to the first query operation includes: displaying a control for viewing the complete trajectory during the display of the target trajectory segment; and sending the first data request to the server device in response to a triggering operation on the control; wherein the complete trajectory to which the target trajectory segment belongs is deemed to be the target trajectory.

[0236] Optionally, in an offline retail scenario, the target space is an offline store, which includes shelves containing items for sale. The target movable object's behavior details are order information generated by the target movable object selecting items in the offline store. The first behavior is the act of selecting items. Accordingly, the target behavior includes at least one of: missed payment, unpaid, overpaid, small order, and the act of selecting a designated item. Accordingly, the target movable object is a user entering or exiting the offline store, including consumers and / or supermarket staff, or may also include an autonomous shopping cart or robot that can move freely within the offline store.

[0237] In an optional embodiment, in response to the second query operation, sending a second data request to the server device to request target trajectory segments marked with the target behavior includes:

[0238] Displaying a second query interface, the second query interface including query condition information items and map data of offline stores;

[0239] In response to a triggering operation on shelf information, a location in a store, and / or a camera in the map data, the triggered shelf information, the location in the store, and / or the camera information is filled in as a query condition;

[0240] A second data request is sent to the server device according to the query condition, so that the server device returns the target trajectory segment marked with the target behavior according to the query condition.

[0241] Furthermore, before displaying the target trajectory, the method of this embodiment further includes: determining whether there is a missing portion in the target trajectory; and if there is a missing portion, completing the target trajectory.

[0242] Optionally, an implementation method for completing a target trajectory includes:

[0243] Respond to the user's request to complete the missing part and determine candidate cameras in the target space that may have captured the missing part.

[0244] Playing the video content captured by the candidate camera so that the user can confirm whether the video content contains the target movable object; and

[0245] In response to a completion confirmation operation initiated by the user, the target trajectory is completed using the trajectory segments containing the target movable object in the video content.

[0246] In this embodiment, the trajectory of the movable object is associated with a data object that can reflect the behavior of the movable object in the target space. This can be used as the data basis for video processing based on the movable object. Not only can the trajectory of the movable object be displayed, but also the data objects related to its behavior in the target space can be associated and displayed. The relevant information of the movable object in the target space can be displayed in a multi-dimensional and stereoscopic manner, which can improve the efficiency of video processing and reduce the cost of video processing.

[0247] Figure 9 A flowchart of a video completion method provided by an exemplary embodiment of the present application is shown in FIG. Figure 9 As shown, the method includes:

[0248] 91. Display the target trajectory. The target trajectory is the trajectory of the target movable object in the target space. The target trajectory may have missing parts.

[0249] 92. Based on the direction of the target movable object before the missing part, determine the candidate cameras in the target space that are likely to capture the missing part.

[0250] 93. Complete the target trajectory based on the video content corresponding to the missing time period captured by the candidate camera; the missing time period is the time period corresponding to the missing part.

[0251] In this embodiment, the target space refers to a physical space that accommodates and allows movable objects to enter and exit. A movable object broadly refers to any object that can move, including autonomous objects such as users, robots, or unmanned vehicles, as well as non-autonomous objects such as human-driven vehicles. The specific implementation of the target space will vary depending on the application scenario, and accordingly, the movable objects that can enter and exit the target space will also vary.

[0252] In some application scenarios, the target space is an offline store such as a shopping mall or supermarket. Correspondingly, the movable objects are users entering and exiting the offline store. The users here include consumers and / or supermarket staff, or may also include autonomous shopping carts that can move freely in the offline store, or robots, etc.

[0253] In this embodiment, the target movable object is any movable object that enters and exits the target space, and the trajectory of the target movable object in the target space is obtained in advance. In this embodiment, the method of obtaining the trajectory of the target movable object is not concerned, for example, but not limited to Figure 7a or Figure 7bThe embodiment shown is not limited thereto.

[0254] In an optional embodiment, the method of this embodiment further includes: in the process of displaying the target trajectory, associating and displaying a map of the target space; the map displays a camera in the target space and a dynamic icon, and the dynamic icon is linked to the target movable object in the target trajectory.

[0255] Based on the above, an implementation of step 92 includes: responding to a completion operation initiated by the user for the missing part, calculating the last direction of the target movable object before the missing part; adjusting the orientation of the camera that last photographed the target movable object before the missing part on the map to be consistent with the last direction; responding to a selection operation initiated by the user for a camera within the orientation coverage range, determining the camera selected by the user as a candidate camera.

[0256] Further optionally, before responding to a selection operation initiated by a user for a camera within the coverage area, it also includes: marking the camera that has not captured the target movable object on the map, so that the user can initiate a selection operation for the camera that is within the coverage area and has not captured the target movable object.

[0257] In an optional embodiment, before displaying the target trajectory, the method further includes: determining a missing portion in the target trajectory and its corresponding missing time period based on the time when the target movable object enters and exits the target space and the time when the target movable object appears in the target trajectory; and marking the missing time period on the timeline of the target trajectory during display of the target trajectory. Accordingly, responding to a user-initiated completion operation for the missing portion includes: responding to a user triggering an operation for the missing time period on the timeline, determining that the user has initiated a completion operation for the missing portion.

[0258] In an optional embodiment, an implementation of step 93 includes: playing the video content captured by the candidate camera corresponding to the time period to be displayed, so that the user can check whether it contains the target movable object; and responding to a completion confirmation operation initiated by the user, completing the target trajectory using a trajectory segment containing the target movable object; wherein the time period to be displayed at least includes the missing time period.

[0259] Further optionally, playing the video content captured by the candidate camera corresponding to the time period to be displayed includes: playing the specified video content captured by the candidate camera, marking the missing time period on the timeline of the specified video content; responding to the user's operation of selecting a time point forward and backward from the missing time period, determining the starting time point and ending time point of the time period to be displayed; starting from the starting time point, playing the video content captured by the candidate camera until the end time point.

[0260] In this embodiment, for a target trajectory with a missing portion, the direction of the movable object before the missing portion can be combined to determine the candidate cameras that are likely to have captured the missing portion; then, the target trajectory is completed based on the video content captured by the candidate cameras, which can reduce the video content that needs to be screened and help improve the completion efficiency.

[0261] It should be noted that the execution entity of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution entity of steps 91 to 93 can be device A; for another example, the execution entity of steps 91 and 92 can be device A, and the execution entity of step 93 can be device B; and so on.

[0262] In addition, some of the processes described in the above embodiments and the accompanying drawings include multiple operations that appear in a specific order, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or may be executed in parallel. The sequence numbers of the operations, such as 91, 92, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0263] Figure 10a This is a schematic diagram of the structure of a server device provided by an exemplary embodiment of the present application. Figure 10a As shown, the server device includes: a memory 103a, a processor 101a and a communication component 102a.

[0264] The memory 103a is used to store computer programs and can be configured to store various other data to support operations on the server device. Examples of such data include instructions for any application or method operating on the server device, video data, images, messages, etc.

[0265] The processor 101a is coupled to the memory 103a and is configured to execute a computer program in the memory 103a, so as to: obtain video data captured by multiple cameras in a target space, where the target space includes at least one movable object; merge the video data captured by the multiple cameras in units of movable objects to obtain a trajectory of the at least one movable object; and associate the trajectory of the at least one movable object with at least one data object, where the at least one data object is related to the behavior of the movable object in the target space.

[0266] In an optional embodiment, when the processor 101a associates the trajectory of at least one movable object with at least one data object, it is specifically used to: for a first movable object, associate the trajectory of the first movable object with the time data of the first movable object entering and exiting the target space, the behavior details data of the first movable object in the target space and / or the map data of the target space; wherein the first movable object is any movable object among the at least one movable object.

[0267] Optionally, when associating the trajectory of the first movable object with the time data of the first movable object entering and exiting the target space, the processor 101a is specifically configured to: generate time axis information of the trajectory of the first movable object according to the time data of the first movable object entering and exiting the target space.

[0268] Optionally, when associating the trajectory of the first movable object with the map data of the target space, the processor 101a is specifically configured to: add a dynamic icon to the map data of the target space based on the trajectory of the first movable object, where the dynamic icon is linked to the first movable object. For example, the dynamic icon can be an avatar registered online by the first movable object.

[0269] Optionally, when the processor 101a associates the trajectory of the first movable object with the behavior details data of the first movable object in the target space, it is specifically used to: determine the behavior object, behavior position and / or behavior time of the first behavior of the first movable object based on the behavior details data of the first movable object in the target space; establish a correspondence between the behavior object, behavior position and / or behavior time of the first behavior and the trajectory segment where the first behavior is located in the trajectory of the first movable object.

[0270] In an optional embodiment, the processor 101a is further configured to: analyze, based on the trajectory of the first movable object, whether the first movable object has or may have a second behavior; and if so, add marking information related to the second behavior to the trajectory of the first movable object.

[0271] Optionally, when the processor 101a adds marking information related to the second behavior to the trajectory of the first movable object, it is specifically configured to perform at least one of the following operations:

[0272] Adding a highlight mark to the behavior object of the second behavior in the trajectory of the first movable object;

[0273] Adding at least one of detailed information of the behavior object of the second behavior, online sales information, offline sales information, offline inventory, and replenishment suggestion information to the trajectory of the first movable object;

[0274] Adding a highlight mark for the behavior time of the second behavior in the trajectory of the first movable object;

[0275] Adding a highlight mark for the behavior position of the second behavior in the trajectory of the first movable object;

[0276] In the trajectory of the first movable object, adding a highlight mark for the first movable object;

[0277] In the track of the first movable object, detailed information of the first movable object is added.

[0278] In an optional embodiment, the target space is an offline store, which contains items for sale; the behavior detail data is the order information formed by the first movable object selecting items in the offline store; the first behavior is the behavior of selecting items; accordingly, the second behavior includes: at least one of: missed payment behavior, unpaid behavior, overpayment behavior, small order behavior and behavior of purchasing designated items.

[0279] In an optional embodiment, the processor 101a is further used to: receive a second data request sent by the display terminal through the communication component 102a, the second data request including second identification information pointing to the target behavior, the target behavior belonging to the second behavior; obtain a target trajectory segment containing the target behavior from the trajectory of at least one movable object according to the second identification information; and provide the target trajectory segment and the marking information related to the target behavior in the target trajectory segment to the display terminal, so that the display terminal can associate and display the target trajectory segment and the marking information related to the target behavior contained therein.

[0280] In an optional embodiment, the processor 101a is further used to: receive a first data request sent by the display terminal through the communication component 102a, the first data request including first identification information pointing to the target trajectory; obtain the target trajectory from the trajectory of at least one movable object based on the first identification information; and provide the target trajectory and at least one data object associated therewith to the display terminal, so that the display terminal can associate and display the target trajectory and the at least one data object associated therewith.

[0281] Optionally, the first identification information is an identification of the target movable object, or an identification of behavior detail data of the target movable object; the target movable object is a movable object corresponding to the target track.

[0282] In an optional embodiment, when the processor 101a obtains the trajectory of at least one movable object, it is specifically used to: perform target tracking on the video data captured by each of the multiple cameras to obtain the movable objects captured by each of the multiple cameras; perform pedestrian re-identification REID processing on the movable objects captured by each of the multiple cameras to obtain the trajectory of at least one movable object.

[0283] Optionally, when the processor 101a obtains the movable objects captured by multiple cameras, it is specifically used to: perform a target tracking operation for each camera every N frames of video data, and in each target tracking operation, combine the N frames of video data to determine whether the movable object tracked in the previous target tracking operation is tracked; where N is an integer greater than or equal to 2.

[0284] Optionally, when the processor 101a performs REID processing on the movable objects captured by each of the multiple cameras to obtain the trajectory of at least one movable object, it is specifically used to: combine the overlapping fields of view between the multiple cameras, the spatiotemporal information of the movable objects and / or the feature information of the movable objects obtained by other means, and perform REID processing on the movable objects captured by each of the multiple cameras to obtain the trajectory of at least one movable object.

[0285] Further optionally, when the processor 101a obtains the characteristic information of the movable object by other means, it is specifically used to perform at least one of the following operations: obtaining the facial image of the movable object photographed by the POS machine in the target space; obtaining the facial image in the electronic account corresponding to the movable object.

[0286] Optionally, the movable object in this embodiment may be a user, a robot, or an autonomous shopping cart in the target space, but is not limited thereto.

[0287] Further, if Figure 10a As shown, the server device also includes: a power supply component 104a and other components. Figure 10a Only some components are shown schematically, which does not mean that the server device only includes Figure 10a Components shown.

[0288] Accordingly, the present invention also provides a computer-readable storage medium storing a computer program, which can achieve the above-mentioned Figure 7a or Figure 7b Each step in the method embodiment.

[0289] Figure 10b This is a schematic diagram of the structure of a display terminal provided by an exemplary embodiment of the present application. Figure 10b As shown, the display terminal includes: a memory 103b, a processor 101b, a communication component 102b and a display 104b.

[0290] The memory 103b is used to store computer programs and can be configured to store various other data to support operations on the display terminal. Examples of such data include instructions for any application or method used to operate on the display terminal, video data, pictures, messages, contact information, etc.

[0291] The processor 101b is coupled to the memory 103b and is configured to execute a computer program in the memory 103b, so as to: in response to a first query operation, send a first data request to the server device via the communication component 102b to request a target trajectory; the target trajectory is a trajectory of a target movable object within a target space; receive the target trajectory and at least one data object associated with the target trajectory returned by the server device; and during the target trajectory display process, display the at least one data object in association with the display 104b, wherein the at least one data object is related to the behavior of the target movable object within the target space.

[0292] In an optional embodiment, the first data request carries first identification information pointing to the target trajectory, where the first identification information is an identification of a target movable object associated with the target trajectory, an identification of behavior detail data of the target movable object, or time data of entering and exiting the target space.

[0293] In an optional embodiment, the at least one data object includes: time data of the target movable object entering and exiting the target space, detailed behavior data of the target movable object in the target space and / or map data of the target space.

[0294] Optionally, when the processor 101b displays the time data of the target movable object entering and exiting the target space through the display 104b, it is specifically configured to: display the time data of the target movable object entering and exiting the target space on the time axis of the target trajectory.

[0295] Optionally, the above-mentioned behavior details data includes: the behavior object, behavior location and / or behavior time of the first behavior of the target movable object in the target space. Based on this, when the processor 101b associates and displays the behavior details data of the target movable object in the target space, it is specifically configured to perform at least one of the following operations:

[0296] On the time axis of the target trajectory, the behavior time when the target movable object takes the first behavior in the target space is displayed;

[0297] Displaying, in an information display area outside the target trajectory display area, a behavior object of the target movable object performing a first behavior in the target space;

[0298] A behavior position where the target movable object performs a first behavior in the target space is marked in the map data.

[0299] Further optionally, when the processor 101b marks the behavior position where the target movable object performs the first behavior in the target space in the map data, it is specifically used to: when a trajectory segment containing the behavior object is displayed, mark the behavior position where the first behavior occurs in the trajectory segment in the map data.

[0300] Optionally, when the processor 101b displays the map data of the target space through the display 104b, it is specifically used to: display the map data of the target space during the target trajectory display process; and display a dynamic icon in the map data, the dynamic icon being linked to the target movable object in the target trajectory.

[0301] Further optionally, when displaying the map data of the target space, the processor 101b is specifically configured to: display the map data of the target space in an information display area outside the target trajectory display area; or display the map data of the target space in a floating layer above the target trajectory display area.

[0302] In an optional embodiment, the processor 101b is further configured to perform at least one of the following operations:

[0303] When the track segment containing the behavior object is displayed through the display 104b, the behavior object is highlighted in the information display area;

[0304] When the track segment containing the behavior object is displayed on the display 104b, the behavior time at which the first behavior of the behavior object occurs is highlighted on the time axis;

[0305] In response to a triggering operation on a behavior object in the information display area, jumping from a current display position to a trajectory segment containing the behavior object;

[0306] In response to the triggering operation of any behavior time on the timeline, jump from the current display position to the track segment corresponding to the triggered behavior time.

[0307] In an optional embodiment, the map data includes icons of multiple cameras in the target space. Based on this, the processor 101b is further configured to: during the target trajectory display process, follow the movement of the target movable object and dynamically mark the camera that captured the target movable object in the map data.

[0308] In an optional embodiment, the processor 101b is further used to: respond to the second query operation, send a second data request to the server device through the communication component 102b to request a target trajectory segment marked with the target behavior; receive the target trajectory segment returned by the server device and the marking information related to the target behavior in the target trajectory segment; and during the display process of the target trajectory segment, associate and display the marking information related to the target behavior in the target trajectory segment through the display 104b.

[0309] Based on the above optional embodiment, when the processor 101b sends the first data request to the server device through the communication component 102b, it is specifically configured to: display a control for viewing the complete trajectory through the display 104b during the display of the target trajectory segment; and in response to a triggering operation on the control, send the first data request to the server device through the communication component 102b; wherein the complete trajectory to which the target trajectory segment belongs is regarded as the target trajectory.

[0310] Optionally, in an offline retail scenario, the target space is an offline store, which includes shelves containing items for sale; the behavior detail data is order information formed by the first movable object selecting items in the offline store; the first behavior is the behavior of selecting items; accordingly, the target behavior includes: at least one of: missed payment behavior, unpaid behavior, overpayment behavior, small order behavior, and behavior of purchasing designated items.

[0311] In an optional embodiment, when the processor 101b sends a second data request to the server device through the communication component 102b, it is specifically used to: display a second query interface through the display 104b, the second query interface including query condition information items and map data of offline stores; respond to triggering operations on shelf information, store locations and / or cameras in the map data, and fill in the triggered shelf information, store locations and / or camera information as query conditions; send a second data request to the server device according to the query conditions, so that the server device returns the target trajectory fragment marked with the target behavior according to the query conditions.

[0312] Furthermore, the processor 101b is further configured to: before displaying the target trajectory, determine whether there is a missing portion in the target trajectory; if there is a missing portion, complete the target trajectory.

[0313] Optionally, when completing the target trajectory, the processor 101b is specifically used to: respond to the completion operation initiated by the user for the missing part, and determine the candidate cameras in the target space that may have captured the missing part; play the video content captured by the candidate cameras for the user to confirm whether the video content contains the target movable object; and respond to the completion confirmation operation initiated by the user, and use the trajectory segment containing the target movable object in the video content to complete the target trajectory.

[0314] Further, if Figure 10b As shown, the server device also includes: a power supply component 105b, an audio component 106b and other components. Figure 10b Only some components are shown schematically, which does not mean that the display terminal only includes Figure 10b Components shown.

[0315] Accordingly, the present invention also provides a computer-readable storage medium storing a computer program, which can achieve the above-mentioned Figure 8 Each step in the method embodiment.

[0316] Figure 10c This is a schematic diagram of another display terminal provided by an exemplary embodiment of the present application. Figure 10c As shown, the display terminal includes: a memory 103c, a processor 101c, a communication component 102c and a display 104c.

[0317] The memory 103c is used to store computer programs and can be configured to store various other data to support operations on the display terminal. Examples of such data include instructions for any application or method used to operate on the display terminal, video data, pictures, messages, contact information, etc.

[0318] The processor 101c is coupled to the memory 103c and is configured to execute a computer program in the memory 103c, so as to: display a target trajectory via the display 104c, where the target trajectory is a trajectory of a target movable object within a target space, and where a missing portion exists in the target trajectory; determine, based on a direction of the target movable object before the missing portion, candidate cameras within the target space that are likely to have captured the missing portion; and complete the target trajectory based on video content captured by the candidate cameras that corresponds to the missing time period; the missing time period is a time period corresponding to the missing portion.

[0319] In an optional embodiment, the processor 101c is further used to: in the process of displaying the target trajectory through the display 104c, associate and display a map of the target space; the map displays a camera in the target space and a dynamic icon, and the dynamic icon is linked to the target movable object in the target trajectory.

[0320] Optionally, when determining the candidate cameras that may have captured the missing part in the target space, the processor 101c is specifically used to: respond to the completion operation initiated by the user for the missing part, calculate the last direction of the target movable object before the missing part; adjust the orientation of the camera that last captured the target movable object before the missing part on the map to be consistent with the last direction; respond to the selection operation initiated by the user for the cameras within the orientation coverage range, and determine the camera selected by the user as the candidate camera.

[0321] Further optionally, the processor 101c is also used to: mark the cameras that have not captured the target movable object on the map before responding to the user's selection operation initiated for the cameras within the coverage area, so that the user can initiate a selection operation for the cameras that are within the coverage area and have not captured the target movable object.

[0322] In an optional embodiment, the processor 101c is further configured to: before displaying the target trajectory via the display 104c, determine the missing portion and its corresponding missing time period in the target trajectory based on the time when the target movable object enters and exits the target space and the time when the target movable object appears in the target trajectory; and during display of the target trajectory, mark the missing time period on the timeline of the target trajectory. Accordingly, when responding to a user-initiated completion operation for the missing portion, the processor 101c is specifically configured to: respond to the user's triggering operation for the missing time period on the timeline and determine that the user has initiated a completion operation for the missing portion.

[0323] In an optional embodiment, when the processor 101c is completing the target trajectory, it is specifically used to: play the video content corresponding to the time period to be displayed captured by the candidate camera through the display 104c, so that the user can check whether it contains the target movable object; and respond to the completion confirmation operation initiated by the user, using the trajectory segment containing the target movable object to complete the target trajectory; wherein the time period to be displayed at least includes the missing time period.

[0324] Further optionally, when the processor 101c plays the video content corresponding to the time period to be displayed and captured by the candidate camera through the display 104c, it is specifically used to: play the specified video content captured by the candidate camera through the display 104c, and mark the missing time period on the time axis of the specified video content; respond to the user's operation of selecting a time point forward and backward from the missing time period, and determine the starting time point and the ending time point of the time period to be displayed; starting from the starting time point, play the video content captured by the candidate camera until the end time point.

[0325] Further, if Figure 10c As shown, the server device also includes: a power supply component 105c, an audio component 106c and other components. Figure 10c Only some components are shown schematically, which does not mean that the display terminal only includes Figure 10c Components shown.

[0326] Accordingly, the present invention also provides a computer-readable storage medium storing a computer program, which can achieve the above-mentioned Figure 9 Each step in the method embodiment.

[0327] The memory in the above embodiments can be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0328] The communication component in each of the above embodiments is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0329] The display in each of the above embodiments includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0330] The power supply assembly in each of the above embodiments provides power to various components of the device in which the power supply assembly is located. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply assembly is located.

[0331] The audio components in the above embodiments may be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal may be further stored in a memory or sent via a communication component. In some embodiments, the audio component further includes a speaker for outputting an audio signal.

[0332] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0333] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0334] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0335] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0336] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0337] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0338] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0339] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0340] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A video processing system, characterized in that: include: Multiple cameras, server devices, and display terminals deployed in the target space; The target space allows movable objects to enter and exit; The multiple cameras are used to capture video data within their respective fields of view and upload the captured video data to the server device; The server device is configured to merge the video data of the plurality of cameras in units of movable objects to obtain a trajectory of each movable object entering the target space, and associate the trajectory of each movable object with at least one data object, wherein the at least one data object is related to the behavior of the movable object in the target space; The display terminal is configured to obtain a target trajectory and at least one data object associated therewith from the server device according to a query operation, and in the process of displaying the target trajectory, display the at least one data object associated therewith; the target trajectory is a trajectory of the target movable object in the target space; In which, the at least one data object includes detailed behavior data of the movable object in the target space; when the server-side device associates the trajectory of each movable object with at least one data object, it is specifically used to: determine the behavior object, behavior position and / or behavior time of the first movable object performing the first behavior based on the detailed behavior data of the first movable object in the target space; establish a correspondence between the behavior object, behavior position and / or behavior time of the first behavior and the trajectory segment where the first behavior is located in the trajectory of the first movable object; analyze whether the first movable object has or may have a second behavior based on the trajectory of the first movable object, and the second behavior is other behavior different from the first behavior; if so, add marking information related to the second behavior to the trajectory of the first movable object; wherein the first movable object is any movable object among the movable objects.

2. The system according to claim 1, wherein: The target space is an offline store, and the movable object is a user entering and exiting the offline store.

3. The system according to claim 2, characterized in that The at least one data object includes: the time when the user enters and exits the offline store, the order details of the user in the offline store and / or a map of the offline store.

4. A video processing method, characterized in that: include: Acquiring video data captured by multiple cameras in a target space, where the target space includes at least one movable object; Merging the video data captured by the multiple cameras in units of movable objects to obtain a trajectory of the at least one movable object; Associating the trajectory of the at least one movable object with at least one data object, the at least one data object being related to the behavior of the movable object in the target space; In which, the at least one data object includes detailed behavior data of the movable object in the target space; when associating the trajectory of the at least one movable object with the at least one data object respectively, it includes: determining the behavior object, behavior position and / or behavior time of the first movable object performing the first behavior based on the detailed behavior data of the first movable object in the target space; establishing a correspondence between the behavior object, behavior position and / or behavior time of the first behavior and the trajectory segment where the first behavior is located in the trajectory of the first movable object; analyzing whether the first movable object has or may have a second behavior based on the trajectory of the first movable object, and the second behavior is other behavior different from the first behavior; if so, adding marking information related to the second behavior to the trajectory of the first movable object; wherein the first movable object is any movable object among the at least one movable object.

5. The method according to claim 4, characterized in that Associating the trajectory of the at least one movable object with at least one data object respectively, further comprising: For the first movable object, the trajectory of the first movable object is associated with the time data of the first movable object entering and exiting the target space and / or the map data of the target space.

6. The method according to claim 5, characterized in that Associating the trajectory of the first movable object with time data of the first movable object entering and exiting the target space includes: Time axis information of a trajectory of the first movable object is generated according to time data of the first movable object entering and exiting the target space.

7. The method according to claim 5, characterized in that Associating the trajectory of the first movable object with map data of the target space includes: According to the trajectory of the first movable object, a dynamic icon is added to the map data of the target space, and the dynamic icon is linked with the first movable object.

8. The method according to claim 7, characterized in that The dynamic icon is an avatar of the first movable object registered online.

9. The method according to claim 4, characterized in that Adding marking information related to the second behavior to the trajectory of the first movable object includes at least one of the following operations: Adding a highlight mark to the behavior object of the second behavior in the trajectory of the first movable object; Adding at least one of detailed information of the behavior object of the second behavior, online sales information, offline sales information, offline inventory, and replenishment suggestion information to the trajectory of the first movable object; Adding a highlight mark for the behavior time of the second behavior in the trajectory of the first movable object; Adding a highlight mark for the behavior position of the second behavior in the trajectory of the first movable object; Adding a highlight mark for the first movable object in the trajectory of the first movable object; Detailed information of the first movable object is added to the trajectory of the first movable object.

10. The method according to claim 4, characterized in that The target space is an offline store, and the offline store contains items for sale; The behavior detail data is order information formed by the first movable object purchasing items in the offline store; The first behavior is an act of purchasing an item; The second behavior includes at least one of: missed payment, unpaid, overpaid, small order, and purchase of designated items.

11. The method according to claim 4, characterized in that Also includes: receiving a second data request sent by the display terminal, wherein the second data request includes second identification information pointing to a target behavior, and the target behavior belongs to the second behavior; acquiring, according to the second identification information, a target trajectory segment containing a target behavior from the trajectory of the at least one movable object; The target trajectory segment and the mark information related to the target behavior in the target trajectory segment are provided to a display terminal, so that the display terminal associates the target trajectory segment and the mark information related to the target behavior contained therein with each other for display.

12. The method according to any one of claims 4 to 11, characterized in that: Also includes: receiving a first data request sent by a display terminal, wherein the first data request includes first identification information pointing to a target track; acquiring the target trajectory from the trajectory of the at least one movable object according to the first identification information; The target trajectory and at least one data object associated therewith are provided to the display terminal, so that the display terminal displays the target trajectory and the at least one data object associated therewith in an associated manner.

13. The method according to claim 11, characterized in that The first identification information is an identification of a target movable object, or an identification of behavior detail data of the target movable object; the target movable object is a movable object corresponding to the target track.

14. The method according to any one of claims 4 to 11, characterized in that: The method of merging the video data captured by the plurality of cameras in units of movable objects to obtain a trajectory of the at least one movable object includes: Performing target tracking on the video data captured by each of the multiple cameras to obtain movable objects captured by each of the multiple cameras; Perform pedestrian re-identification (REID) processing on the movable objects captured by each of the multiple cameras to obtain a trajectory of the at least one movable object.

15. The method according to claim 14, characterized in that Performing target tracking on the video data captured by each of the multiple cameras to obtain movable objects captured by each of the multiple cameras includes: For each camera, a target tracking operation is performed every N frames of video data. In each target tracking operation, the N frames of video data are combined to determine whether the movable object tracked in the previous target tracking operation is tracked; where N is an integer greater than or equal to 2.

16. The method according to claim 15, characterized in that Performing pedestrian re-identification (REID) processing on the movable objects captured by each of the multiple cameras to obtain a trajectory of the at least one movable object includes: Combined with the overlapping fields of view between the multiple cameras, the spatiotemporal information of the movable objects and / or the feature information of the movable objects obtained by other means, REID processing is performed on the movable objects photographed by each of the multiple cameras to obtain the trajectory of the at least one movable object.

17. The method according to claim 16, characterized in that Obtaining feature information of the movable object by other means, including at least one of the following: Acquiring a facial image of a movable object captured by a POS terminal in the target space; Obtain a facial image from an electronic account corresponding to a movable object.

18. The method according to any one of claims 4 to 11, characterized in that: The movable object is a user, a robot, or an autonomous shopping cart in the target space.

19. A server device, characterized in that: include: memory and processor; The memory is used to store computer programs; The processor, coupled to the memory, is configured to execute the computer program for: Acquiring video data captured by multiple cameras in a target space, where the target space includes at least one movable object; Merging the video data captured by the multiple cameras in units of movable objects to obtain a trajectory of the at least one movable object; Associating the trajectory of the at least one movable object with at least one data object, the at least one data object being related to the behavior of the movable object in the target space; In which, the at least one data object includes detailed behavior data of the movable object in the target space; when associating the trajectory of the at least one movable object with the at least one data object respectively, it includes: determining the behavior object, behavior position and / or behavior time of the first movable object performing the first behavior based on the detailed behavior data of the first movable object in the target space; establishing a correspondence between the behavior object, behavior position and / or behavior time of the first behavior and the trajectory segment where the first behavior is located in the trajectory of the first movable object; analyzing whether the first movable object has or may have a second behavior based on the trajectory of the first movable object, and the second behavior is other behavior different from the first behavior; if so, adding marking information related to the second behavior to the trajectory of the first movable object; wherein the first movable object is any movable object among the at least one movable object.

20. A trajectory display method, characterized in that: include: In response to the first query operation, a first data request is sent to the server device to request a target trajectory; The target trajectory is the trajectory of the target movable object in the target space; receiving a target trajectory and at least one data object associated with the target trajectory returned by the server device; During the target trajectory display process, the at least one data object is displayed in association, the at least one data object being related to the behavior of the target movable object in the target space; In which, the at least one data object includes detailed behavior data of the movable object in the target space; the step of associating the target trajectory with the at least one data object includes: determining the behavior object, behavior position and / or behavior time of the first behavior of the target movable object based on the detailed behavior data of the target movable object in the target space; establishing a correspondence between the behavior object, behavior position and / or behavior time of the first behavior and the trajectory segment where the first behavior is located in the trajectory of the target movable object; analyzing whether the target movable object has or may have a second behavior based on the trajectory of the target movable object, and the second behavior is other behavior different from the first behavior; if so, adding marking information related to the second behavior to the trajectory of the target movable object.

21. The method according to claim 20, characterized in that The first data request carries first identification information pointing to the target trajectory, where the first identification information is an identification of a target movable object associated with the target trajectory, an identification of behavior detail data of the target movable object, or time data of entering and exiting the target space.

22. The method according to claim 20, characterized in that The at least one data object includes: time data of the target movable object entering and exiting the target space and / or map data of the target space.

23. The method according to claim 22, characterized in that During the target trajectory display process, the associated display of the time data of the target movable object entering and exiting the target space includes: On the time axis of the target trajectory, time data of the target movable object entering and exiting the target space is displayed.

24. The method according to claim 22, characterized in that The behavior detail data includes: the behavior object, behavior location and / or behavior time of the first behavior of the target movable object in the target space; During the target trajectory display process, the associated display of the behavior detail data of the target movable object in the target space includes at least one of the following operations: On the time axis of the target trajectory, display the behavior time when the target movable object performs a first behavior in the target space; Displaying, in an information display area outside the target trajectory display area, a behavior object of the target movable object performing a first behavior in the target space; A behavior position where the target movable object performs a first behavior in the target space is marked in the map data.

25. The method according to claim 24, characterized in that Marking, in the map data, a behavior position where the target movable object performs a first behavior in the target space, includes: When a track segment containing the behavior object is displayed, a behavior position where the first behavior occurs in the track segment is marked in the map data.

26. The method according to claim 22, characterized in that In the target trajectory display process, the map data of the target space is displayed in association, including: During the target trajectory display process, map data of the target space is displayed; and a dynamic icon is displayed in the map data, wherein the dynamic icon is linked to the target movable object in the target trajectory.

27. The method according to claim 26, characterized in that During the target trajectory display process, displaying map data of the target space includes: The map data of the target space is displayed in an information display area outside the target track display area; or the map data of the target space is displayed in a floating layer above the target track display area.

28. The method according to claim 24, characterized in that Also includes at least one of the following operations: When a trajectory segment containing the behavior object is displayed, highlighting the behavior object in the information display area; When the trajectory segment containing the behavior object is displayed, the behavior time when the first behavior of the behavior object occurs is highlighted on the time axis; In response to a triggering operation on the behavior object in the information display area, jumping from a current display position to a trajectory segment containing the behavior object; In response to a triggering operation on any behavior time on the timeline, the display jumps from the current display position to the track segment corresponding to the triggered behavior time.

29. The method according to claim 26, wherein The map data includes icons of multiple cameras in the target space, and the method further includes: During the target trajectory display process, the movement of the target movable object is followed, and the camera that photographed the target movable object is dynamically marked in the map data.

30. The method according to claim 20, wherein Also includes: In response to the second query operation, sending a second data request to the server device to request the target trajectory segment marked with the target behavior; receiving a target trajectory segment and marking information related to the target behavior in the target trajectory segment returned by the server device; During the display of the target trajectory segment, marking information related to the target behavior in the target trajectory segment is displayed in association.

31. The method according to claim 30, wherein In response to the first query operation, sending a first data request to the server device includes: During the display of the target trajectory segment, a control for viewing the complete trajectory is displayed; In response to a triggering operation on the control, a first data request is sent to the server device; wherein the complete trajectory to which the target trajectory segment belongs is the target trajectory.

32. The method according to claim 30, wherein The target space is an offline store, which includes shelves containing items for sale; The behavior detail data is order information formed by the target movable object selecting items in the offline store; The first behavior is an act of purchasing an item; The target behavior includes at least one of: missed payment, unpaid, overpaid, small order, and purchase of designated items.

33. The method according to claim 32, characterized in that In response to the second query operation, a second data request is sent to the server device to request target trajectory segments marked with the target behavior, including: Displaying a second query interface, wherein the second query interface includes query condition information items and map data of the offline store; In response to a triggering operation on shelf information, a location in a store, and / or a camera in the map data, filling in the triggered shelf information, the location in the store, and / or the camera information as a query condition; A second data request is sent to the server device according to the query condition, so that the server device returns the target trajectory segment marked with the target behavior according to the query condition.

34. The method according to any one of claims 20 to 33, characterized in that Before displaying the target trajectory, the method further includes: Determining whether there is a missing portion in the target trajectory; If there are missing parts, the target trajectory is completed.

35. The method according to claim 34, wherein Completing the target trajectory includes: In response to a completion operation initiated by a user for the missing portion, determining a candidate camera in the target space that is likely to capture the missing portion; Playing the video content captured by the candidate camera to allow the user to confirm whether the video content contains the target movable object; and In response to a completion confirmation operation initiated by the user, the target trajectory is completed using the trajectory segment containing the target movable object in the video content.

36. A display terminal, characterized in that: include: memory, processors, communication components, and displays; The memory is used to store computer programs; The processor, coupled to the memory, is configured to execute the computer program to: In response to the first query operation, a first data request is sent to the server device to request a target trajectory; the target trajectory is a trajectory of the target movable object in the target space; receiving a target trajectory and at least one data object associated with the target trajectory returned by the server device; During the target trajectory display process, the at least one data object is displayed in association, the at least one data object being related to the behavior of the target movable object in the target space; In which, the at least one data object includes detailed behavior data of the movable object in the target space; the step of associating the target trajectory with the at least one data object includes: determining the behavior object, behavior position and / or behavior time of the first behavior of the target movable object based on the detailed behavior data of the target movable object in the target space; establishing a correspondence between the behavior object, behavior position and / or behavior time of the first behavior and the trajectory segment where the first behavior is located in the trajectory of the target movable object; analyzing whether the target movable object has or may have a second behavior based on the trajectory of the target movable object, and the second behavior is other behavior different from the first behavior; if so, adding marking information related to the second behavior to the trajectory of the target movable object.

37. A trajectory completion method, characterized in that: include: Displaying a target trajectory, wherein the target trajectory is a trajectory of a target movable object in a target space, the target trajectory being generated by the video processing method according to any one of claims 4 to 18, and the target trajectory having missing parts; determining, based on the direction of the target movable object before the missing portion, candidate cameras in the target space that are likely to capture the missing portion; Completing the target trajectory according to the video content corresponding to the missing time period captured by the candidate camera; The missing time period is a time period corresponding to the missing portion.

38. The method according to claim 37, wherein Also includes: In the process of displaying the target trajectory, a map of the target space is displayed in association; The map displays a camera in the target space and a dynamic icon, and the dynamic icon is linked to the target movable object in the target track.

39. The method according to claim 38, characterized in that Determining, based on the direction of the target movable object before the missing portion, candidate cameras in the target space that may capture the missing portion, including: In response to a completion operation initiated by a user for the missing portion, calculating a final direction of the target movable object before the missing portion; Adjusting the direction of the camera on the map that last captured the target movable object before the missing portion to be consistent with the final direction; In response to a selection operation initiated by a user with respect to a camera within the coverage area, the camera selected by the user is determined as the candidate camera.

40. The method according to claim 39, wherein Before responding to a selection operation initiated by a user on a camera within the coverage area, the method further includes: The cameras that have not captured the target movable object are marked on the map, so that the user can initiate a selection operation for the cameras that are within the coverage range and have not captured the target movable object.

41. The method according to claim 39, wherein Before displaying the target track, it also includes: determining a missing portion in the target trajectory and its corresponding missing time period based on a time when the target movable object enters and exits the target space and a time when the target movable object appears in the target trajectory; and In the process of displaying the target trajectory, the missing time period is marked on the time axis of the target trajectory.

42. The method according to claim 41, wherein Responding to a completion operation initiated by the user for the missing part includes: In response to a user triggering an operation on a missing time period on the timeline, it is determined that the user initiates a completion operation on the missing portion.

43. The method according to any one of claims 28 to 42, characterized in that The target trajectory is completed according to the video content corresponding to the missing time period captured by the candidate camera, including: Playing the video content captured by the candidate camera and corresponding to the time period to be displayed, so that the user can check whether the video content contains the target movable object; and In response to a completion confirmation operation initiated by the user, the target trajectory is completed using a trajectory segment containing the target movable object; wherein the time period to be displayed at least includes the missing time period.

44. The method according to claim 43, wherein Playing the video content captured by the candidate camera and corresponding to the time period to be displayed includes: Play the specified video content captured by the candidate camera, and mark the missing time period on the timeline of the specified video content; In response to the user selecting a time point forward and backward from the missing time period, determining the start time point and the end time point of the time period to be displayed; Starting from the starting time point, the video content captured by the candidate camera is played until the ending time point.

45. A display terminal, characterized in that: include: memory, processors, communication components, and displays; The memory is used to store computer programs; The processor, coupled to the memory, is configured to execute the computer program to: Displaying a target trajectory, wherein the target trajectory is a trajectory of a target movable object in a target space, the target trajectory being generated by the video processing method according to any one of claims 4 to 18, and the target trajectory having missing parts; determining, based on the direction of the target movable object before the missing portion, candidate cameras in the target space that are likely to capture the missing portion; Completing the target trajectory according to the video content corresponding to the missing time period captured by the candidate camera; The missing time period is a time period corresponding to the missing portion.

46. ​​A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by one or more processors, the one or more processors are caused to implement the steps of the method according to any one of claims 4 to 18, 20 to 35 and 37 to 44.

Images