Data processing method and related apparatus

Abstract

This invention proposes a data processing method and related apparatus. By dividing each image frame into multiple sub-regions and using the camera index number of the captured image frame and the sub-region index number corresponding to the sub-region where the target object is located, the position of the target object is marked, thereby obtaining a position index sequence corresponding to the video sequence. This scheme of storing the target object's movement trajectory through an index sequence reduces storage burden and subsequent computational processing burden, improving the processing efficiency of trajectory prediction. By dividing each image frame into multiple sub-regions and locating the target object on a sub-region basis, the position prediction of the target object can be achieved at a finer granular level, and image information unrelated to the target object can be excluded from the image frame. This scheme, combining fine granularity and filtering out irrelevant information for target object localization, can improve the prediction accuracy of movement trajectories.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically, to a data processing method and related apparatus. Background Technology

[0002] The continuous expansion of video surveillance has generated massive amounts of video data. Consequently, when it is necessary to locate a specific target object in a video, it is necessary to manually search through the surveillance videos of the target object and use experience to estimate the target object's movement trajectory based on the surveillance videos.

[0003] However, this method of manually searching for surveillance videos and estimating their trajectories not only requires a lot of time to sift through the videos and is inefficient in locating the target, but also cannot accurately predict the target's movement trajectory. Summary of the Invention

[0004] In view of this, in order to at least solve the technical problem of the inability to accurately predict the movement trajectory of a target object in related technologies, the purpose of this invention is to provide a data processing method and related apparatus. The technical solution adopted in the embodiments of this invention is as follows:

[0005] A first aspect of the present invention provides a data processing method, comprising:

[0006] Obtain the video sequence and access control sequence of the target object for a continuous duration up to and including the current moment;

[0007] Obtain the camera index number corresponding to each image frame in the video sequence, divide each image frame into multiple sub-regions, and obtain the sub-region index number corresponding to the target sub-region where the target object is located in each image frame;

[0008] Based on the camera index number and sub-region index number corresponding to each image frame, a position index sequence corresponding to the video sequence is obtained;

[0009] From the pre-stored list of access control device indexes, obtain the access control index code corresponding to the unique identifier of each device in the access control sequence, so as to obtain the access control index sequence corresponding to the access control sequence;

[0010] The location index sequence and the access control index sequence are time-aligned to obtain the trajectory index sequence;

[0011] The trajectory index sequence is numerically encoded to obtain a fused trajectory sequence;

[0012] Based on the fused trajectory sequence, a pre-trained trajectory prediction model is used to process the target object to obtain the predicted trajectory sequence at the next moment.

[0013] The predicted trajectory sequence is decoded to obtain a predicted trajectory index sequence; the predicted trajectory index sequence is used to indicate the position of the target object at the next moment.

[0014] A second aspect of the present invention provides a data processing apparatus, comprising:

[0015] The acquisition module is configured to acquire video sequences and access control sequences of the target object for a continuous duration up to and including the current time.

[0016] The index processing module is configured to: obtain the camera index number corresponding to each image frame in the video sequence, divide each image frame into multiple sub-regions, and obtain the sub-region index number corresponding to the target sub-region where the target object is located in each image frame;

[0017] The location index module is configured to: obtain a location index sequence corresponding to the video sequence based on the camera index number and sub-region index number corresponding to each image frame;

[0018] The access control index module is configured to: retrieve the access control index code corresponding to the unique identifier of each device in the access control sequence from a pre-stored access control device index list, so as to obtain the access control index sequence corresponding to the access control sequence;

[0019] The alignment module is configured to perform time alignment processing on the location index sequence and the access control index sequence to obtain a trajectory index sequence;

[0020] The encoding module is configured to perform numerical encoding processing on the trajectory index sequence to obtain a fused trajectory sequence;

[0021] The prediction module is configured to: process the fused trajectory sequence based on the pre-trained trajectory prediction model to obtain the predicted trajectory sequence of the target object at the next moment;

[0022] The decoding module is configured to decode the predicted trajectory sequence to obtain a predicted trajectory index sequence; the predicted trajectory index sequence is used to indicate the position of the target object at the next moment.

[0023] A third aspect of the present invention provides an electronic device including a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor can execute the machine-executable instructions to implement the data processing method provided in the first aspect above.

[0024] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the data processing method provided in the first aspect described above.

[0025] The data processing method and related apparatus provided in this invention, on the one hand, divide each image frame into multiple sub-regions, and use the camera index number of the captured image frame and the sub-region index number corresponding to the sub-region where the target object is located in the image frame to mark the position of the target object, thereby obtaining a position index sequence corresponding to the video sequence. It can be seen that the temporally continuous data in the position index sequence represents the movement trajectory of the target object. This scheme of storing the movement trajectory of the target object through the index sequence can not only reduce the storage burden, but also reduce the subsequent computational processing burden, thereby improving the processing efficiency of predicting the movement trajectory of the target object at the next moment. Furthermore, by dividing each image frame into multiple sub-regions and locating the target object on a sub-region basis, compared with locating the target object with an entire image, the position prediction of the target object can be achieved at a finer granular level, and image information unrelated to the target object in the image frame can be excluded. Thus, this scheme of locating the target object by combining fine granularity and filtering out irrelevant information can achieve more precise positioning of the target object, thereby making the subsequent prediction of the movement trajectory at the next moment based on this precise positioning data have higher prediction accuracy. On the other hand, by fusing video data and access control data to record and predict the movement trajectory of a target object, it is possible to locate the target object and obtain its movement trajectory even if the target object is not within the monitoring range of any camera. Furthermore, since there is a certain spatial correlation between the time-aligned fused video data and access control data, using this fused data to record and predict the movement trajectory of the target object can further improve the prediction accuracy of the target object's movement trajectory in the next moment.

[0026] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0027] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This diagram illustrates a structural block diagram of an electronic device provided by an embodiment of the present invention.

[0029] Figure 2 A flowchart of a data processing method provided by an embodiment of the present invention is shown;

[0030] Figure 3 A functional block diagram of a data processing device provided in an embodiment of the present invention is shown. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0032] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0033] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0034] To address the technical problem of inaccurate prediction of target object movement trajectories in related technologies, this invention provides a data processing method. On one hand, by dividing each image frame into multiple sub-regions and using the camera index number of the captured image frame and the sub-region index number corresponding to the target object's location within the image frame, the position of the target object is marked, thereby obtaining a position index sequence corresponding to the video sequence. It is evident that the temporally continuous data in the position index sequence represents the target object's movement trajectory. This scheme of storing the target object's movement trajectory through an index sequence not only reduces storage burden but also reduces subsequent computational processing burden, thus improving the processing efficiency for predicting the target object's movement trajectory at the next moment. Furthermore, by dividing each image frame into multiple sub-regions and locating the target object on a sub-region basis, compared to locating the target object using an entire image, a more granular prediction of the target object's position can be achieved. It also eliminates image information unrelated to the target object from the image frame. Therefore, this scheme, combining granularity and filtering out irrelevant information for target object location, can achieve more precise target object location, resulting in higher prediction accuracy for the next moment's movement trajectory predicted based on this precise location data. On the other hand, by fusing video data and access control data to record and predict the movement trajectory of a target object, it is possible to locate the target object and obtain its movement trajectory even if the target object is not within the monitoring range of any camera. Furthermore, since there is a certain spatial correlation between the time-aligned fused video data and access control data, using this fused data to record and predict the movement trajectory of the target object can further improve the prediction accuracy of the target object's movement trajectory in the next moment.

[0035] The data processing method provided by this invention can be applied to electronic devices. Please refer to [link / reference]. Figure 1 This is a structural block diagram of an electronic device. The electronic device 100 includes a memory 110, a processor 120, and a communication module 130. The memory 110, processor 120, and communication module 130 are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0036] The memory is used to store programs or data. The memory may be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc.

[0037] The processor is used to read / write data or programs stored in memory and to perform the corresponding functions.

[0038] The communication module is used to establish communication connections between electronic devices and other communication terminals via a network, and to send and receive data via the network.

[0039] It should be understood that, Figure 1 The structure shown is only a schematic diagram of an electronic device; the electronic device may also include components that are larger than... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown. Figure 1 The components shown can be implemented using hardware, software, or a combination thereof.

[0040] In some embodiments, the electronic device may be a device with computing capabilities, such as at least one of a smartwatch, smartphone, personal computer, or server. The data processing method provided in this embodiment of the invention may be loaded into the electronic device as a software program or assembled into the electronic device as a hardware functional module. This embodiment of the invention does not limit the specific implementation of this method.

[0041] In some embodiments, the electronic device can be applied in the field of video surveillance. For example, it can serve as a monitoring center that communicates with each camera and access control device in the monitored area, and is used to process the video sequences captured by the cameras and the access control sequences collected by the access control devices to achieve tracking of target objects and prediction of their movement trajectories.

[0042] The following combination Figure 2 The data processing method provided in the embodiments of the present invention will be described below. Figure 2 This is a flowchart of a data processing method provided in an embodiment of the present invention. The data processing method provided in this embodiment of the present invention includes:

[0043] In step S100, the video sequence and access control sequence of the target object for a continuous set duration at the current time and before are obtained;

[0044] In step S200, the camera index number corresponding to each image frame in the video sequence is obtained, and each image frame is divided into multiple sub-regions. The sub-region index number corresponding to the target sub-region where the target object is located in each image frame is obtained.

[0045] In step S300, a position index sequence corresponding to the video sequence is obtained based on the camera index number and sub-region index number corresponding to each image frame;

[0046] In step S400, the access control index code corresponding to the unique identifier of each device in the access control sequence is obtained from the pre-stored access control device index list to obtain the access control index sequence corresponding to the access control sequence;

[0047] In step S500, the location index sequence and the access control index sequence are time-aligned to obtain the trajectory index sequence;

[0048] In step S600, the trajectory index sequence is numerically encoded to obtain a fused trajectory sequence;

[0049] In step S700, the predicted trajectory sequence of the target object at the next moment is obtained by processing the fused trajectory sequence based on the pre-trained trajectory prediction model.

[0050] In step S800, the predicted trajectory sequence is decoded to obtain a predicted trajectory index sequence; the predicted trajectory index sequence is used to indicate the position of the target object at the next moment.

[0051] The data processing method provided in this embodiment of the invention can be applied in a monitoring center, which is connected to each camera and each access control device in the monitored area to receive video sequences collected in real time by each camera and access control sequences collected in real time by the access control devices.

[0052] In the above, the monitored area can be any area that needs to be monitored, such as an industrial park, a residential area, or a school, but is not limited to these. The video sequence can include multiple images, the camera index number corresponding to each image, and the capture time of each image. Each access control data point in the access control sequence can include an identity ID, a unique identifier for the access control device, and the recording time. It is understood that when a person passes through the access control device for verification, regardless of whether the verification is successful, the access control device will retain a record of that person's presence.

[0053] Based on this, when it is necessary to track the trajectory of a target object in the monitored area, the target object to be tracked can be manually input first. For example, the identity ID or facial image of the target object can be entered in the monitoring center, or the target object can be circled in the image screen with the target object captured, so as to determine the target object to be tracked.

[0054] Subsequently, the data processing method provided in this embodiment of the invention is executed. First, step S100 is executed to obtain a video sequence and access control sequence of the target object for a continuously set duration up to and including the current time. The set duration can be set based on experience or requirements, and this embodiment of the invention does not limit this. During the execution of step S100, a face recognition algorithm or target localization algorithm in related technologies can be used to intelligently filter out image frames containing the target object from all video data for a continuously set duration up to and including the current time, based on the input image of the target object, and store the filtered image frames in chronological order, thereby obtaining the video sequence recorded in step S100. In addition, if the target object is an object with a known identity ID, access control data with the same identity ID as the target object can be intelligently filtered out from all access control data for a continuously set duration up to and including the current time, based on the target object's identity ID, and stored in chronological order, thereby obtaining the access control sequence recorded in step S100.

[0055] In some embodiments, to obtain the access control sequence of a target object with an unknown identity ID, the access control device can be an access control device with a camera function. Based on this, the access control device can detect and record access control data for any person passing through it. The access control data may also include the person's facial image information. In this case, by comparing the facial features of the target object, access control data with facial features matching the target object's facial features can be intelligently filtered from all access control data for a continuously set time period prior to the current moment. The filtered access control data is then stored in chronological order, thereby obtaining the access control sequence recorded in step S100.

[0056] After obtaining the video sequence and access control sequence of the target object within a continuously set time period at the current moment through any of the above embodiments, in order to reduce the amount of data processing and achieve more granular and accurate recording of the target object's position, steps S200 to S400 are executed. Steps S200 and S400 can be executed in parallel or in sequence. In the case of serial execution, the order of execution between the two is not limited.

[0057] During step S200, each image frame in the video sequence will be processed as follows:

[0058] Obtain the camera index number of the camera that captured the image; for example, assuming the monitored area has... A camera can then... As the index number of the camera, .

[0059] The image is divided into multiple sub-regions according to a set division rule. This division can be either evenly divided into m sub-regions or unequally divided into m sub-regions, where m is an integer greater than 1. Based on this, the sub-region index numbers can be set sequentially according to their positional relationship: .

[0060] In some examples, to better reduce the difficulty of segmentation and computational complexity while achieving accurate positioning of the target object, m can be 4. This allows for accurate differentiation of the target object's location from four different directions, and since it is divided into only four sub-regions, it does not impose a computational burden on the system. Based on this, if the target object is in the current image frame... If there are several sub-regions, then the sub-region index number of the target object in the current image frame is: ,in, .

[0061] For example, an image can be divided into four sub-regions: top left, top right, bottom left, and bottom right. The sub-region indexes of these four sub-regions can be as follows: Therefore, the quotation marks are now... .

[0062] Therefore, through step S200, the camera index number and sub-region index number corresponding to each image frame can be obtained.

[0063] In some examples, the segmentation rules for each image are the same to ensure that the positioning standards for the target object are consistent across different images, thereby better guaranteeing the accuracy of subsequent trajectory prediction and reducing computational complexity.

[0064] After obtaining the camera index number and sub-region index number corresponding to each image frame through any embodiment in step S200, step S300 is executed to obtain a position index sequence corresponding to the video sequence based on the camera index number and sub-region index number corresponding to each image frame. For example, for each image frame in the video sequence, the image frame can be replaced with an index code formed by combining the camera index number and the sub-region index number, and the time corresponding to the index code can be recorded to obtain an index code that corresponds one-to-one with each image in the video sequence, thereby obtaining a position index sequence.

[0065] Although the location index sequence can be obtained through the above method, in some embodiments, to ensure that the index codes in the location index sequence are arranged sequentially in time order to facilitate subsequent time alignment processing and ensure the accuracy of fused data and the precision of trajectory prediction, the data processing method provided by this embodiment of the invention also provides another scheme for obtaining the location index sequence. That is, in step S300 above, the step of obtaining the location index sequence corresponding to the video sequence based on the camera index number and sub-region index number corresponding to each image frame includes:

[0066] In step S310, the position index code of the target object in each image frame is determined according to the camera index number and sub-region index number corresponding to each image frame; the position index code records the camera index number and the sub-region index number.

[0067] In step S320, the position index code corresponding to each image frame is recorded sequentially according to the time order of each image frame in the video sequence to obtain the position index sequence.

[0068] During step S310, a location index code can be generated based on the camera index number and sub-region index number corresponding to each image frame. For example, assuming the camera index number is... The sub-region index number is The generated position index encoding can then be: .

[0069] Next, step S320 is executed, recording the position index code corresponding to each image frame in the video sequence according to the time sequence of each image frame, thereby forming a position index sequence. The time sequence corresponding to each position index code in the position index sequence can be obtained from the time sequence of each image frame in the video sequence.

[0070] Therefore, based on the location index encoding, it is possible to know at what time which sub-region of which camera's image the target object is in, thus achieving spatial and temporal positioning of the target object.

[0071] During step S400, since a list of access control device indexes is pre-stored, and this list records a one-to-one mapping between the unique identifier of each access control device and its access control index code, the corresponding access control index code can be found in the list based on the unique identifier of each device recorded in the access control sequence. This yields the access control index code corresponding to each unique identifier in the access control sequence, thus obtaining the access control index sequence corresponding to the access control sequence. The timing of each access control index code in the access control index sequence can be determined from the time information recorded in the corresponding access control sequence.

[0072] In some examples, the access control index code can be represented as ,in, This represents the index number of the access control device. For example, suppose there is... Each access control device, .

[0073] It should be understood that different cameras have different camera index numbers, and different access control devices also have different index numbers.

[0074] After obtaining the location index sequence and access control index sequence, step S500 is executed to perform time alignment processing on the location index sequence and access control index sequence to obtain the trajectory index sequence, which is the fused data.

[0075] As described above, time alignment techniques from related technologies can be used to achieve time alignment between the location index sequence and the access control index sequence. However, since the location index sequence is obtained from a high-frequency video sequence, while the access control index sequence is obtained from a low-frequency access control sequence, the data density of the location index sequence is higher than that of the access control index sequence. If time alignment techniques from related technologies are used, the accuracy of the fused data may be affected due to time sequence differences, thereby affecting the accuracy of subsequent trajectory prediction. Therefore, to solve this technical problem, in some embodiments, the step S500 above, which involves time alignment of the location index sequence and the access control index sequence to obtain the trajectory index sequence, includes:

[0076] In step S510, the time alignment range corresponding to each location index code in the location index sequence is determined;

[0077] In step S520, if there is a target access control index code with time overlap within the time alignment range of the currently processed location index code, the currently processed location index code is associated with the target access control index code to obtain the corresponding associated index code;

[0078] In step S530, if there is no time-overlapping target access control index code within the time alignment range of the currently processed location index code, the currently processed location index code is associated with a set index code to obtain a corresponding associated index code; wherein, the set index code is used to characterize that only the location index code exists in the associated index code;

[0079] In step S540, after all location index codes have been processed, all the obtained associated index codes are arranged in chronological order to obtain the trajectory index sequence.

[0080] During step S500, step S510 is executed first. Based on the video timestamp, the time alignment range corresponding to each location index code is obtained according to the time corresponding to each location index code in the location index sequence and the set time difference threshold. For example, assuming the location index code... The corresponding time is The time difference threshold is Then the position index encoding The corresponding time alignment range is Based on this principle, the time alignment range corresponding to each location index code can be obtained. The time difference threshold can be set according to experience or requirements, for example, 2 seconds, but is not limited to this.

[0081] Next, based on the time alignment range corresponding to each location index code, the time alignment range to which each access control index code falls is determined according to the time corresponding to each access control index code in the access control index sequence. For example, suppose we are currently analyzing whether there is a location index code... Access control index codes with overlapping time alignment ranges are identified during the analysis process. The corresponding time t0 falls within the time alignment range Then the position index encoding is considered to be Target access control index encoding with time overlap within the time alignment range Next, step S520 is executed to associate the currently processed location index code with the target access control index code to obtain the corresponding associated index code. For example, the obtained associated index code can be represented as: The associated index encoding can not only represent the spatial location of the target object, but also reflect the spatial correlation between the camera and the access control device. The spatial correlation can indicate that the actual physical distance between the camera and the access control device is relatively close.

[0082] Conversely, if the times corresponding to all access control index codes do not fall within the time alignment range... Then the position index encoding is considered to be If there is no time overlap within the target access control index code within the time alignment range, then step S530 is executed to associate the currently processed position index code with the set index code to obtain the corresponding associated index code. The set index code is used to indicate that only the position index code exists in the associated index code. As an example, the set index code can be configured to -1, but is not limited to this. Based on this example, the associated index code obtained after step S530 can be represented as: This indicates that the fused data contains only video data and no access control data; in other words, the associated index encoding... The indicated fusion data contains video data of the target object, but no corresponding access control data, indicating that the target object has not passed through any access control device, or that it is at the location... Within the set distance range, there are no access control devices. The set distance range can be set according to actual needs and experience, and is not limited here.

[0083] Therefore, after completing the time alignment of all location index codes through step S530 and / or step S540, step S540 is executed to arrange all the obtained associated index codes in chronological order to obtain the trajectory index sequence.

[0084] As an example, a sequence of trajectory indexes can be represented as { , , , , , , , , , , , , , However, this is not the only possibility. It is evident that each associated index code in the trajectory index sequence can represent the location of the target object at the corresponding time. Since these location points are continuous in time, the resulting trajectory index sequence is equivalent to recording the target object's movement trajectory within the current time and the preceding set duration.

[0085] After obtaining the trajectory index sequence fused with video data and access control data through any embodiment in step S500, in order to enable the pre-trained trajectory prediction model to process the trajectory index sequence to obtain the predicted trajectory sequence of the target object at the next moment, step S600 is executed to first perform numerical encoding processing on the trajectory index sequence to obtain the fused trajectory sequence represented in numerical form.

[0086] As described above, the numerical value of each associated index code in the trajectory index sequence can be obtained by querying a pre-constructed spatial mapping vocabulary. The spatial mapping vocabulary stores a one-to-one mapping relationship between each associated index code (formed by combining each location index code and each access control index code) and its corresponding numerical value. Numerical processing techniques, such as tokenizer tools, can be used to generate the numerical value corresponding to each associated index code, and then each associated index code is mapped one-to-one with its corresponding numerical value to obtain the aforementioned spatial mapping vocabulary.

[0087] Therefore, by executing step S600, the trajectory index sequence can be numerically encoded into a fused trajectory sequence. For example, suppose the trajectory index sequence is { , , , , , , , , , , , , , If we consider the fusion trajectory sequence {4539, 4086, 1195, 43, 180, 723, 4851, 3089, 2070, 4515, 3905, 3231, 2385, 1324}, then the fusion trajectory sequence can be {4539, 4086, 1195, 43, 180, 723, 4851, 3089, 2070, 4515, 3905, 3231, 2385, 1324}.

[0088] After obtaining the fused trajectory sequence, step S700 is executed, inputting the fused trajectory sequence at the current moment into the trajectory prediction model to obtain the predicted trajectory sequence of the target object at the next moment. The predicted trajectory sequence and the fused trajectory sequence have the same format, both storing numerical values.

[0089] After obtaining the predicted trajectory sequence, in order to locate the target object and make the predicted trajectory of the target object intuitive and visible, step S800 is executed to decode the predicted trajectory sequence to obtain the predicted trajectory index sequence.

[0090] During step S800, the associated index code corresponding to each value in the predicted trajectory sequence can be obtained by querying the aforementioned spatial mapping vocabulary. These associated index codes are then arranged sequentially according to time, thus obtaining the predicted trajectory index sequence. It is evident that the predicted trajectory index sequence stores the position of the target object for the specified duration up to and including the next moment. Therefore, the predicted trajectory index sequence can be used to determine which sub-region of the image from which camera the target object will appear, and which access control device it will be located at or near. This allows supervisory personnel to promptly determine whether to issue a warning to the target object based on its trajectory at the next moment.

[0091] In some embodiments, to avoid misjudgments of warnings due to fatigue or lack of experience of regulatory personnel, the data processing method provided in this embodiment of the invention also provides two intelligent warning schemes, which can improve the accuracy of warning judgments, as follows:

[0092] The first intelligent early warning solution:

[0093] After obtaining the predicted trajectory sequence, the data processing method provided in this embodiment of the invention may further include:

[0094] In step S910, the correlation between the predicted trajectory sequence and the fused trajectory sequence is calculated to obtain a correlation sequence;

[0095] In step S920, if the minimum value in the correlation sequence is less than the set correlation threshold, the target object is determined to be an abnormal object, and a warning message is output.

[0096] After obtaining the predicted trajectory sequence, in order to achieve early warning of abnormal target objects, step S910 is first executed to calculate the correlation between each predicted value in the predicted trajectory sequence and the corresponding actual value in the fused trajectory sequence, so as to obtain the correlation value between each group of predicted values ​​and actual values ​​and form a correlation sequence.

[0097] In the above, the corresponding predicted value and actual value refer to the fact that the predicted value ranked first in the predicted trajectory sequence corresponds to the actual value ranked first in the fused trajectory sequence, the predicted value ranked second in the pre-stored trajectory sequence corresponds to the actual value ranked second in the fused trajectory sequence, and so on.

[0098] Furthermore, the correlation between the predicted and actual values ​​can be obtained through correlation calculation methods in related technologies, or through a pre-trained trajectory correlation prediction model. Alternatively, the trajectory prediction model can be trained with additional trajectory correlation calculation capabilities to enable it to perform both trajectory prediction and trajectory correlation calculation.

[0099] In some embodiments, to improve the computational efficiency of correlation and thus the efficiency of early warning processing, the trajectory prediction model can be a transformer model. Based on this, the correlation sequence can be obtained by processing the predicted trajectory sequence and the fused trajectory sequence using the attention formula of the trajectory prediction model.

[0100] After obtaining the relevance sequence through any of the above embodiments, the minimum value is extracted from the relevance sequence. Then, the minimum value is compared with the relevance threshold. If the minimum value is greater than or equal to the relevance threshold, it indicates that the target object's movement trajectory at the next moment is within the reachable range of the current moment's movement trajectory and is considered a normal movement trajectory. In this case, the target object is considered a normal object, and no warning is issued. The reachable range refers to the maximum distance the target object can walk from its current moment's movement trajectory within the time interval between the current moment and the next moment, at the normal walking speed of most people.

[0101] Conversely, if the minimum value is less than the relevance threshold, it means that the target object's movement trajectory in the next moment exceeds the reachable range of the movement trajectory in the current moment, and the target object's movement is abnormal. At this time, step S920 is executed to identify the target object as an abnormal object and output warning information.

[0102] Therefore, by using the current movement trajectory to determine whether the movement trajectory in the next moment is normal, the overall trajectory can be used to determine whether the target's actions are suspicious, which can improve the accuracy of early warning and the accuracy of identifying abnormal target objects.

[0103] The second intelligent early warning solution:

[0104] After obtaining the predicted trajectory index sequence, the data processing method provided in this embodiment of the invention may further include:

[0105] In step S910', the target position of the target object at the next moment is determined according to the predicted trajectory index sequence;

[0106] In step S920', if the target location is within the warning zone, a warning message is output.

[0107] After obtaining the predicted trajectory sequence, in order to achieve early warning of abnormal target objects, step S910' is first executed to determine the target position of the target object at the next moment based on the predicted trajectory index sequence. For example, the area of ​​the camera, the sub-region of the image screen, and the area where the access control device is located, indicated by the last associated index code in the predicted trajectory index sequence, can be used as the target position of the target object at the next moment; the target area monitored by the sub-region of the camera indicated by the last associated index code can also be used as the target position; the center position of the target area can also be used as the target position; or the location of the access control device can also be used as the target position.

[0108] The aforementioned area can be determined as follows: First, a planar map of the monitored area is pre-constructed using a relevant map building scheme. Then, each camera and its sub-regions, as well as each access control device, are marked on the planar map to obtain the map location information of each camera, its sub-regions, and the access control devices. A correspondence is then established between each location index code and each access control index code and its corresponding map location information. Based on this, the geographical location information associated with the last associated index code in the predicted trajectory index sequence can be determined through this correspondence. The area that exactly surrounds this geographical location information is then considered the aforementioned area. However, this method is not limited to this.

[0109] After obtaining the target location, it can be determined whether the target location falls within a pre-set warning zone. The determination principle can be found in relevant technologies and will not be elaborated here. If the target location is a point, and it falls within the pre-set warning zone, then the target location is considered to be within the warning zone. If the target location is a region, then if there is an overlap between the target location and the warning zone, the target location is considered to be within the warning zone. In this case, step S920' will be executed to output warning information.

[0110] Conversely, if the target location is not within the warning area, no warning is required.

[0111] In some embodiments, to better improve the prediction accuracy of the trajectory prediction model, the data processing method provided in this embodiment of the invention may further include a process of training the trajectory prediction model, wherein the training process includes:

[0112] In step S010, the initial model is iteratively trained using the training sample set until the loss value of the initial model reaches the set expectation, at which point the currently trained initial model is used as the prediction model.

[0113] The training sample set includes multiple sets of historical fusion trajectory sequences that are continuous in time. These multiple sets of historical fusion trajectory sequences serve as positive samples. Each set of historical fusion trajectory sequences includes multiple historical location point values ​​that are continuous in time and have a set duration. Each historical location point value is used to numerically represent the location index code and access control index code aligned to the historical time. The loss value is calculated by the loss function based on the historical fusion trajectory sequence located at the next moment after the current input historical fusion trajectory sequence and the predicted fusion trajectory sequence output by the initial model.

[0114] The above methods for obtaining multiple sets of historical fusion trajectory sequences can be as follows:

[0115] The system iterates through the historical trajectory data of all individuals and activities within the monitored areas. This historical trajectory data includes historical video data and historical access control data for a set historical duration. The set historical duration can be configured based on actual needs or experience.

[0116] Next, the historical video data corresponding to the same person and occurring consecutively in time are organized into a total historical video sequence, and the historical access control data corresponding to the same person and occurring consecutively in time are organized into a total access control sequence. This yields the total historical video sequence and the total access control sequence for each person.

[0117] Then, using the sliding time window set above as described above, the total historical video sequence and the total access control sequence are processed to obtain multiple historical video sequences and multiple historical access control sequences corresponding to each person.

[0118] Next, based on multiple historical video sequences and multiple historical access control sequences corresponding to each person, multiple sets of historical fusion trajectory sequences are obtained and represented numerically. The principle of obtaining the historical fusion trajectory sequences can be found in the section on obtaining fusion trajectory sequences above, and will not be repeated here.

[0119] Therefore, by using numerically fused trajectory sequences to train the initial model, the initial model can more easily understand the fused trajectory sequences, which is conducive to improving training efficiency. Furthermore, since the fused trajectory sequences not only provide overall trajectory features but also location point features, the initial model can learn the global and local features of the trajectory. As a result, the trajectory prediction model trained in the end can obtain more accurate predicted trajectories by taking into account both global and local features, thereby improving the prediction and positioning accuracy of the target object.

[0120] In some embodiments, to enable the trajectory prediction model to learn the discrimination boundary, the training sample set further includes multiple sets of abnormal fused trajectory sequences used as negative samples; the process of obtaining the multiple sets of abnormal fused trajectory sequences includes:

[0121] In step Sa, all historical trajectory index sequences are traversed to determine multiple target location index codes and multiple target access control index codes that do not have a time overlap relationship in all historical trajectory index sequences;

[0122] In step Sb, each target location index code is combined with different target access control index codes to generate multiple anomaly association index codes;

[0123] In step Sc, multiple sets of abnormal fusion trajectory sequences are constructed based on the encoding of multiple abnormal correlation indexes.

[0124] Through steps Sa to Sc, the association index codes that have not appeared in the historical trajectory index sequence can be determined and used as abnormal association index codes. It is understandable that the association index codes that have not appeared in the historical trajectory index sequence are usually because the corresponding cameras and access control devices do not have spatial correlation; therefore, the target object cannot pass through cameras and access control devices that do not have spatial correlation at the same time. Therefore, the location index codes corresponding to cameras without spatial correlation and the access control index codes corresponding to access control devices can be fused to obtain the corresponding abnormal association index codes. Then, by arranging and combining these abnormal association index codes, multiple sets of abnormal fused trajectory sequences can be constructed.

[0125] It is worth noting that the technical features or solutions in any of the above embodiments of the present invention can be combined with each other, as long as there is no contradiction in the combination.

[0126] In order to perform the corresponding steps in the various embodiments and possible modes of the data processing method, an implementation of a data processing apparatus is given below. Optionally, the data processing apparatus may employ the methods described above. Figure 1 The device structure of the electronic device is shown. Further, please refer to... Figure 3 , Figure 3 This is a functional block diagram of a data processing device provided in an embodiment of the present invention. It should be noted that the basic principle and technical effects of the data processing device provided in this embodiment are the same as those in the aforementioned related embodiments. For the sake of brevity, any parts not mentioned in this embodiment can be referred to the corresponding content in the aforementioned related embodiments. The data processing device 300 includes:

[0127] The acquisition module 310 is configured to acquire the video sequence and access control sequence of the target object for a continuous duration up to and including the current time.

[0128] The index processing module 320 is configured to: obtain the camera index number corresponding to each image frame in the video sequence, divide each image frame into multiple sub-regions, and obtain the sub-region index number corresponding to the target sub-region where the target object is located in each image frame;

[0129] The location index module 330 is configured to obtain a location index sequence corresponding to the video sequence based on the camera index number and sub-region index number corresponding to each image frame;

[0130] The access control index module 340 is configured to: retrieve the access control index code corresponding to the unique identifier of each device in the access control sequence from a pre-stored access control device index list, so as to obtain the access control index sequence corresponding to the access control sequence;

[0131] Alignment module 350 is configured to perform time alignment processing on the location index sequence and the access control index sequence to obtain a trajectory index sequence;

[0132] Encoding module 360 ​​is configured to: perform numerical encoding processing on the trajectory index sequence to obtain a fused trajectory sequence;

[0133] Prediction module 370 is configured to: process the fused trajectory sequence based on the pre-trained trajectory prediction model to obtain the predicted trajectory sequence of the target object at the next moment;

[0134] The decoding module 380 is configured to decode the predicted trajectory sequence to obtain a predicted trajectory index sequence; the predicted trajectory index sequence is used to indicate the position of the target object at the next moment.

[0135] In some embodiments, the process by which the location index module 330 obtains a location index sequence corresponding to the video sequence based on the camera index number and sub-region index number corresponding to each image frame is configured as follows:

[0136] Based on the camera index number and sub-region index number corresponding to each image frame, the position index code of the target object in each image frame is determined; the position index code records the camera index number and the sub-region index number;

[0137] According to the time sequence of each image frame in the video sequence, the position index code corresponding to each image frame is recorded sequentially to obtain the position index sequence.

[0138] In some embodiments, the process by which the alignment module 350 performs time alignment processing on the location index sequence and the access control index sequence to obtain the trajectory index sequence is configured as follows:

[0139] Determine the time alignment range corresponding to each location index code in the location index sequence;

[0140] If there is a target access control index code with overlapping time within the time alignment range of the currently processed location index code, the currently processed location index code is associated with the target access control index code to obtain the corresponding associated index code;

[0141] If there are no time-overlapping target access control index codes within the time alignment range of the currently processed location index code, the currently processed location index code is associated with a set index code to obtain the corresponding associated index code; wherein, the set index code is used to indicate that only the location index code exists in the associated index code;

[0142] After processing all location index codes, all the obtained associated index codes are arranged in chronological order to obtain the trajectory index sequence.

[0143] In some embodiments of the present invention, the data processing apparatus 300 may further include:

[0144] The training module is configured to iteratively train the initial model using a training sample set until the loss value of the initial model reaches a set expectation, and then use the currently trained initial model as the prediction model.

[0145] The training sample set includes multiple sets of historical fusion trajectory sequences that are continuous in time. These multiple sets of historical fusion trajectory sequences serve as positive samples. Each set of historical fusion trajectory sequences includes multiple historical location point values ​​that are continuous in time and have a set duration. Each historical location point value is used to numerically represent the location index code and access control index code aligned to the historical time. The loss value is calculated by the loss function based on the historical fusion trajectory sequence located at the next moment after the current input historical fusion trajectory sequence and the predicted fusion trajectory sequence output by the initial model.

[0146] In some embodiments, the training sample set further includes multiple sets of abnormal fusion trajectory sequences used as negative samples; the data processing apparatus 300 provided in this embodiment of the invention may further include:

[0147] The negative sample acquisition module is configured to: traverse all historical trajectory index sequences to determine multiple target location index codes and multiple target access control index codes that do not have a time overlap relationship in all historical trajectory index sequences; combine each target location index code with different target access control index codes to generate multiple abnormal association index codes; and construct multiple sets of abnormal fusion trajectory sequences based on the multiple abnormal association index codes.

[0148] In some embodiments, the data processing apparatus 300 provided in this invention may further include: a first early warning module, configured to: after the prediction module 370 processes and obtains the predicted trajectory sequence, calculate the correlation between the predicted trajectory sequence and the fused trajectory sequence to obtain a correlation sequence; if the minimum value in the correlation sequence is less than a set correlation threshold, determine that the target object is an abnormal object and output early warning information.

[0149] In some embodiments of the present invention, the data processing apparatus 300 may further include:

[0150] The second early warning module is configured to: after the decoding module 380 processes and obtains the predicted trajectory index sequence, determine the target position of the target object at the next moment based on the predicted trajectory index sequence; and output early warning information if the target position is in the early warning area.

[0151] Optionally, the above modules can be stored in the form of software or firmware. Figure 1 The memory shown is either stored in or embedded in the operating system (OS) of the electronic device, and can be... Figure 1 The processor executes the commands. Meanwhile, the data and program code required to execute these modules can be stored in memory.

[0152] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0153] In addition, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0154] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0155] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A data processing method, characterized in that, include: Obtain the video sequence and access control sequence of the target object for a continuous duration up to and including the current moment; Obtain the camera index number corresponding to each image frame in the video sequence, divide each image frame into multiple sub-regions, and obtain the sub-region index number corresponding to the target sub-region where the target object is located in each image frame; The location index code of the target object in each image frame is determined based on the camera index number and sub-region index number corresponding to each image frame; The location index code records the camera index number and the sub-region index number; According to the time sequence of each image frame in the video sequence, the position index code corresponding to each image frame is recorded sequentially to obtain the position index sequence; From the pre-stored list of access control device indexes, obtain the access control index code corresponding to the unique identifier of each device in the access control sequence, so as to obtain the access control index sequence corresponding to the access control sequence; Determine the time alignment range corresponding to each location index code in the location index sequence; If there is a target access control index code with overlapping time within the time alignment range of the currently processed location index code, the currently processed location index code is associated with the target access control index code to obtain the corresponding associated index code; If there are no time-overlapping target access control index codes within the time alignment range of the currently processed location index code, the currently processed location index code is associated with a set index code to obtain the corresponding associated index code; wherein, the set index code is used to indicate that only the location index code exists in the associated index code; After processing all location index codes, all the obtained associated index codes are arranged in chronological order to obtain the trajectory index sequence; The trajectory index sequence is numerically encoded to obtain a fused trajectory sequence; Based on the fused trajectory sequence, a pre-trained trajectory prediction model is used to process the target object to obtain the predicted trajectory sequence at the next moment. The predicted trajectory sequence is decoded to obtain a predicted trajectory index sequence; the predicted trajectory index sequence is used to indicate the position of the target object at the next moment.

2. The method according to claim 1, characterized in that, The method further includes a process of training the trajectory prediction model, wherein the training process includes: The initial model is iteratively trained using a training sample set until the loss value of the initial model reaches the set expectation. Then, the currently trained initial model is used as the prediction model. The training sample set includes multiple sets of historical fusion trajectory sequences that are continuous in time. These multiple sets of historical fusion trajectory sequences serve as positive samples. Each set of historical fusion trajectory sequences includes multiple historical location point values ​​that are continuous in time and have a set duration. Each historical location point value is used to numerically represent the location index code and access control index code aligned to the historical time. The loss value is calculated by the loss function based on the historical fusion trajectory sequence located at the next moment after the current input historical fusion trajectory sequence and the predicted fusion trajectory sequence output by the initial model.

3. The method according to claim 2, characterized in that, The training sample set also includes multiple sets of abnormal fusion trajectory sequences used as negative samples; the process of obtaining the multiple sets of abnormal fusion trajectory sequences includes: Traverse all historical trajectory index sequences to determine multiple target location index codes and multiple target access control index codes that do not have time overlap in all historical trajectory index sequences; Each target location index code is combined with a different target access control index code to generate multiple anomaly association index codes; Multiple sets of anomaly fusion trajectory sequences are constructed based on the encoding of multiple anomaly association indexes.

4. The method according to any one of claims 1 to 3, characterized in that, After obtaining the predicted trajectory sequence, the method further includes: Calculate the correlation between the predicted trajectory sequence and the fused trajectory sequence to obtain a correlation sequence; If the minimum value in the relevance sequence is less than the set relevance threshold, the target object is determined to be an abnormal object, and a warning message is output.

5. The method according to any one of claims 1 to 3, characterized in that, After obtaining the predicted trajectory index sequence, the method further includes: The target position of the target object at the next moment is determined based on the predicted trajectory index sequence; If the target location is within the warning zone, a warning message is output.

6. A data processing apparatus, characterized in that, include: The acquisition module is configured to acquire video sequences and access control sequences of the target object for a continuous duration up to and including the current time. The index processing module is configured to: obtain the camera index number corresponding to each image frame in the video sequence, divide each image frame into multiple sub-regions, and obtain the sub-region index number corresponding to the target sub-region where the target object is located in each image frame; The location index module is configured to: determine the location index code of the target object in each image frame based on the camera index number and sub-region index number corresponding to each image frame; the location index code records the camera index number and the sub-region index number; and record the location index code corresponding to each image frame in the time order of each image frame in the video sequence to obtain the location index sequence. The access control index module is configured to: retrieve the access control index code corresponding to the unique identifier of each device in the access control sequence from a pre-stored access control device index list, so as to obtain the access control index sequence corresponding to the access control sequence; The alignment module is configured to: determine the time alignment range corresponding to each location index code in the location index sequence; if there is a target access control index code with time overlap within the time alignment range of the currently processed location index code, associate the currently processed location index code with the target access control index code to obtain a corresponding associated index code; if there is no target access control index code with time overlap within the time alignment range of the currently processed location index code, associate the currently processed location index code with a set index code to obtain a corresponding associated index code; wherein, the set index code is used to characterize that only location index codes exist in the associated index codes; after processing all location index codes, arrange all the obtained associated index codes in chronological order to obtain a trajectory index sequence; The encoding module is configured to perform numerical encoding processing on the trajectory index sequence to obtain a fused trajectory sequence; The prediction module is configured to: process the fused trajectory sequence based on the pre-trained trajectory prediction model to obtain the predicted trajectory sequence of the target object at the next moment; The decoding module is configured to decode the predicted trajectory sequence to obtain a predicted trajectory index sequence; the predicted trajectory index sequence is used to indicate the position of the target object at the next moment.

7. An electronic device, characterized in that, It includes a processor and a memory, the memory storing machine-executable instructions that can be executed by the processor to implement the method of any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 5.

Images