A data processing method, apparatus, device, and storage medium

Abstract

This application discloses a data processing method, apparatus, and device, comprising: determining a first region in a first image when no target object is detected in the first image; the first image is an image obtained by image acquisition of the current scene; the first region is the region occupied by the interactive object of the target object in the first image; at least calling a region estimation model to estimate the first region to obtain a target search region, so as to re-search for the target object in the target search region. This scheme improves the accuracy of the determined target search region and further improves the success rate of re-searching after the target object is lost.

Description

Technical Field

[0001] This application relates to the field of target tracking technology, and includes, but is not limited to, a data processing method, apparatus, device, and storage medium. Background Technology

[0002] With the continuous development of image processing technology, it has been widely used in various fields. Target tracking based on image processing technology has also received widespread attention.

[0003] During the tracking of a target object, the target object may be lost due to movement or turning, thus requiring a re-search.

[0004] For re-searching after losing a target object, related technologies generally establish a search box with a fixed threshold centered on the target object's previous position (before losing the target object), and use this search box as the search area for re-searching the target object within this search area.

[0005] In practice, since the target object may move after being lost, the search area determined by a fixed threshold has a high probability of failure when re-searching. Summary of the Invention

[0006] This application provides a data processing method, apparatus, device, and storage medium. This solution improves the accuracy of the target search area obtained after the target object is lost, and further improves the success rate of re-search.

[0007] The technical solution of this application is implemented as follows:

[0008] In a first aspect, this application provides a data processing method, the method comprising:

[0009] If no target object is detected in the first image, a first region is determined in the first image; the first image is an image obtained by capturing images of the current scene; the first region is the area occupied by the interactive object of the target object in the first image;

[0010] The region estimation model is invoked to estimate at least the first region to obtain the target search region, so as to re-search the target object in the target search region.

[0011] Secondly, this application provides a data processing apparatus, the apparatus comprising:

[0012] The determining unit is configured to determine a first region in the first image when no target object is detected in the first image; the first image is an image obtained by image acquisition of the current scene; the first region is the region occupied by the interactive object of the target object in the first image;

[0013] An estimation unit is used to invoke a region estimation model to perform estimation processing on at least the first region to obtain a target search region, so as to perform a re-search on the target object in the target search region.

[0014] Thirdly, this application also provides an electronic device, including: a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the program to implement the above-described data processing method.

[0015] Fourthly, this application also provides a storage medium on which a computer program is stored, which, when executed by a processor, implements the above-described data processing method.

[0016] The data processing method, apparatus, device, and storage medium provided in this application include: determining a first region in a first image when no target object is detected in the first image; the first image is an image obtained by image acquisition of the current scene; the first region is the region occupied by the interactive object of the target object in the first image; and calling a region estimation model to estimate at least the first region to obtain a target search region, so as to re-search for the target object in the target search region.

[0017] In the scheme of this application, if the target object is not detected in the first image, i.e., the target object is lost, the first region occupied by the target object's interaction object in the first image is first determined, and the region estimation model is called to estimate the first region to obtain the target search region. It can be seen that: firstly, when determining the target search region, the influence of the target object's interaction object on the search region is considered, which improves the accuracy of the search region and the success rate of the search; and secondly, the target search region is automatically generated by the region estimation model, which has a higher degree of automation and adaptability. Attached Figure Description

[0018] Figure 1 A schematic diagram of an optional flow of the data processing system provided in an embodiment of this application;

[0019] Figure 2 A schematic diagram of a first optional data processing method provided in an embodiment of this application;

[0020] Figure 3 A schematic diagram of a second optional data processing method provided in an embodiment of this application;

[0021] Figure 4 A schematic diagram of a third optional data processing method provided in the embodiments of this application;

[0022] Figure 5 A schematic diagram of a fourth optional data processing method provided in the embodiments of this application;

[0023] Figure 6 A schematic diagram of a fifth optional data processing method provided in the embodiments of this application;

[0024] Figure 7 A schematic diagram of a sixth optional data processing method provided in the embodiments of this application;

[0025] Figure 8 A schematic diagram of an optional scenario for the segmentation process provided in an embodiment of this application;

[0026] Figure 9 This is a schematic diagram of an optional structure of the image segmentation algorithm provided in the embodiments of this application;

[0027] Figure 10 A schematic diagram of an optional process for providing behavior detection results in an embodiment of this application;

[0028] Figure 11 This is a schematic diagram of an optional structure of the behavior detection algorithm provided in the embodiments of this application;

[0029] Figure 12 This is a schematic diagram of an optional structure of the search region estimation model provided in an embodiment of this application;

[0030] Figure 13 This is a schematic diagram of an optional structure of the data processing apparatus provided in an embodiment of this application;

[0031] Figure 14 This is a schematic diagram of an optional structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the specific technical solutions of the application will be further described in detail below with reference to the accompanying drawings of the embodiments of this application. The following embodiments are used to illustrate this application, but are not intended to limit the scope of this application.

[0033] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0034] In the following description, the terms "first," "second," and "third" are used only to distinguish different objects and do not represent a specific order of objects, nor are they constituting a chronological order. It is understood that "first," "second," and "third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0035] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0036] This application provides data processing methods, apparatus, devices, and storage media. In practical applications, the data processing method can be implemented by a data processing apparatus, and the functional entities in the data processing apparatus can be collaboratively implemented by the hardware resources of electronic devices (such as terminal devices), such as computing resources like processors and communication resources (such as those used to support various communication methods like optical fiber and cellular networks).

[0037] The data processing method provided in this application is applied to a data processing system. The data processing system includes an image acquisition end and a data processing end. The image acquisition end and the data processing end can interact with each other. Wherein:

[0038] The image acquisition terminal is used to acquire images of the current scene.

[0039] The data processing terminal is used to execute the data processing method provided in the embodiments of this application, that is, to track the target object based on the image acquired by the image acquisition terminal.

[0040] It should be noted that the image acquisition end and the data processing end can be deployed independently on different electronic devices; or, the image acquisition end and the data processing end can be integrated on the same electronic device.

[0041] In one example, such as Figure 1 As shown, the data processing system 10 may include a camera 101 and a processor 102.

[0042] The camera 101 is used to: capture the current scene to obtain a first image.

[0043] This application does not limit the specific type of camera 101, and it can be configured according to actual conditions. For example, camera 101 may include: bullet camera, pan-tilt camera, dome camera, PTZ camera, etc.

[0044] The processor 102 is configured to perform the following: if no target object is detected in the first image, determine a first region in the first image; the first image is an image obtained by image acquisition of the current scene; the first region is the region occupied by the interactive object of the target object in the first image; invoke a region estimation model to estimate at least the first region to obtain a target search region, so as to re-search for the target object in the target search region.

[0045] The processor 102 is an electronic device with relevant data processing capabilities.

[0046] Below, in conjunction with Figure 1 The schematic diagram shown illustrates various embodiments of the data processing method, apparatus, device, and storage medium provided in this application.

[0047] Firstly, embodiments of this application provide a data processing method, which is applied to a data processing apparatus. Exemplarily, the method is applied to... Figure 1 The processor 102 in the electronic device. The function implemented by this method can be achieved by the processor in the electronic device calling program code. Of course, the program code can be stored in the computer storage medium. It can be seen that the electronic device includes at least a processor and a storage medium.

[0048] The following describes the data processing method provided in the embodiments of this application, using an electronic device as the execution subject as an example.

[0049] This data processing method is used to re-search for a target object after it has been lost. Figure 2 This is a flowchart illustrating the data processing method according to an embodiment of this application, as shown below. Figure 2 As shown, the process may include, but is not limited to, S201 and S202 described below.

[0050] S201. If no target object is detected in the first image of the electronic device, a first region is determined in the first image.

[0051] The first image is an image obtained by capturing images of the current scene; the first region is the region occupied by the interactive object of the target object in the first image.

[0052] The target object refers to the object being tracked by the electronic device. This application does not specifically limit the type of target object; it can be configured according to actual circumstances. For example, the target object can be a person, an animal, or a moving object (toy car, toy airplane), etc.

[0053] The electronic device acquires an image of the current scene to obtain a first image, then identifies the target object in the first image. If the target object is not identified in the first image, that is, the target object is lost, then a first region is determined in the first image.

[0054] Here, the first image is an image in which the target object cannot be identified. This application does not specifically limit the method of acquiring the first image; it can be configured according to actual circumstances. For example, the first image can be acquired in real time by an electronic device. As another example, the first image can also be acquired by an image acquisition terminal and then sent to the electronic device.

[0055] This application embodiment does not limit the current scene and can be configured according to the actual situation. For example, the current scene can be an indoor scene or an outdoor scene, etc. This application embodiment does not limit the objects included in the current scene and can be determined according to the actual situation. For example, the current scene may include: person 1, person 2, table 1, chair 1, chair 2, wall 1, etc.

[0056] The process of determining the first region in the first image will now be explained.

[0057] The first region is the first region occupied by the interactive object of the target object in the first image.

[0058] The interaction object of the target object refers to the object that has a certain relationship with the target object in the current scene. For example, if the target object is person 1, the interaction object of the target object can be the chair that person 1 is sitting on; or it can be person 2 who is shaking hands with person 1.

[0059] Specifically, the electronic device first identifies all objects in the first image, determines the interactive object of the target object among all objects, and then determines the area occupied by the interactive object in the first image as the first region.

[0060] S202. The electronic device at least calls the region estimation model to estimate the first region to obtain the target search region, so as to re-search the target object in the target search region.

[0061] The region estimation model is configured to determine the target search region based at least on a first region occupied by the interacting object. This application does not limit the specific structure of the region estimation model; it can be configured according to actual conditions. In one possible implementation, the region estimation model can be a multi-layer neural network. For example, the region estimation model can be a multi-layer perceptron (MLP).

[0062] The embodiments of this application do not limit the training process of the region estimation model, and can be configured according to the actual situation. For example, the neural network can be trained through supervised learning or backpropagation algorithm to obtain the region estimation model.

[0063] The embodiments of this application do not limit the form of the first region and the target search region, and can be configured according to actual conditions. For example, the first region and the target search region can be configured as image formats. The background region is black, and the first region and the target search region correspond to the white regions in the image, respectively. It is understood that the first region and the target search region can also be configured in other forms representing location regions, such as pixel coordinates, etc.

[0064] In one possible implementation, the first region of the target interactive object is used as a factor influencing the target search region. Correspondingly, S202 can be implemented as follows: the electronic device calls the region estimation model to estimate the first region to obtain the target search region, so as to re-search for the target object in the target search region.

[0065] In another possible implementation, the first region of the target interaction object and auxiliary information (the predicted behavior of the target object and / or the category of the interaction object) are used as factors affecting the target search region. Correspondingly, S202 can be implemented as follows: the electronic device calls the region estimation model to estimate the first region to obtain the initial region, and obtains the target search region based on the initial region and its auxiliary features, so as to re-search for the target object in the target search region.

[0066] Specifically, re-searching for the target object in the target search area includes: projecting the target search area (e.g., a polygonal box) onto a first image to obtain a search image in the first image; identifying the target object in the search image; if the identification is successful, the search is determined to be successful; if the identification fails, the re-search is determined to have failed.

[0067] The data processing method provided in this application includes: when no target object is detected in the first image, determining a first region in the first image; the first image is an image obtained by image acquisition of the current scene; the first region is the region occupied by the interactive object of the target object in the first image; calling a region estimation model to estimate at least the first region to obtain a target search region, so as to re-search for the target object in the target search region.

[0068] In the scheme of this application, if the target object is not detected in the first image, i.e., the target object is lost, the first region occupied by the target object's interaction object in the first image is first determined, and the region estimation model is called to estimate the first region to obtain the target search region. It can be seen that: firstly, when determining the target search region, the influence of the target object's interaction object on the search region is considered, which improves the accuracy of the search region and the success rate of the search; and secondly, the target search region is automatically generated by the region estimation model, which has a higher degree of automation and adaptability.

[0069] The process by which the S201 electronic device determines the first region in the first image will now be described.

[0070] In one possible implementation, refer to Figure 3 The process may include, but is not limited to, the following S2011 to S2013.

[0071] S2011. The electronic device uses an image segmentation algorithm to segment the first image to obtain the positions of at least two objects.

[0072] The at least two objects are all objects in the first image.

[0073] This application does not specifically limit the image segmentation algorithm, and it can be configured according to actual conditions. For example, the image segmentation algorithm can be a segmenter.

[0074] For example, S2011 can be implemented as follows: the electronic device uses an image segmentation algorithm to segment objects in the first image to obtain all objects in the first image and the location area occupied by each object.

[0075] S2012. The electronic device identifies the location region of the interactive object within the location regions of the at least two objects.

[0076] In the frame image before the target object was lost, the interaction object of the target object has already been identified. Therefore, the identity characteristics of the interaction object can be obtained. Thus, the interaction object can be identified based on the identity characteristics of the interaction object among at least two objects, and then the location region of the interaction object can be determined within the location regions of at least two objects.

[0077] The following section explains the process of determining the interactive object from the frame image before the tracking is lost.

[0078] The embodiments of this application do not specifically limit the method for determining the interaction object, and can be configured according to the actual situation.

[0079] In one possible implementation, the electronic device determines the distance between each object and the target object (e.g., the target object in the image before it was lost) based on the position of each object in the historical frame image (frame image before it was lost), and identifies the object with the smallest distance from the target object as the interactive object of the target object.

[0080] In another possible implementation, the electronic device determines the intersection of the regions between each object and the target object based on the position of each object in the historical frame image (the frame image before the loss of tracking), and identifies the object with the largest intersection between the regions of each object and the target object as the target object.

[0081] S2013, The electronic device determines the location area of ​​the interactive object as the first area.

[0082] In this way, the objects in the first image are segmented using image segmentation to identify interactive objects, and then the location region of the interactive object is obtained based on the position of each object. It can be seen that this method is simple to implement and reliable.

[0083] The following describes the process by which the electronic device in S202 at least calls the region estimation model to estimate the first region and obtain the target search region.

[0084] This process may include, but is not limited to, method 1 or method 2 below.

[0085] Method 1: Obtain the target search area based on the first region and the region estimation model;

[0086] Method 2: Obtain the target search area based on the first region, auxiliary features, and region estimation model.

[0087] The process of obtaining the target search area based on the first region and the region estimation model in Method 1 will be explained below.

[0088] In Method 1, the region estimation model is configured to determine the target search region based on the first region occupied by the interactive object. Therefore, the region estimation model is called to estimate the first region to obtain the target search region.

[0089] The region estimation model is trained based on multiple first regions in the training set and the target search region corresponding to each first region. The specific training process will not be described here.

[0090] The process of obtaining the target search area based on the first region, auxiliary features, and region estimation model in Method 2 is explained below.

[0091] In one possible implementation, the process may include, but is not limited to, Figure 4S2021 to S2024 are shown.

[0092] S2021. The electronic device calls the first neural network to estimate the first region features representing the first region to obtain the initial region features.

[0093] The first neural network is used to estimate the initial region of the target object.

[0094] For example, the first neural network can be a multi-layer perceptron (MLP).

[0095] The first neural network is configured as a network model that obtains initial region features based on the features of a first region. Here, the initial region features refer to the region features obtained solely from the first region features when determining the target search region, without considering the influence of auxiliary features. The training process of the first neural network will not be described in detail in this embodiment.

[0096] S2021 can be implemented as follows: the electronic device inputs the first region features representing the first region into the first neural network, and estimates the target search region through the first neural network to obtain the initial region features.

[0097] It should be noted that if the first region is obtained in the form of an image, the first region in the form of an image is encoded by an encoder to obtain the first region features before being input into the first neural network. If the first region is obtained in the form of features, the first region features are directly input into the first neural network.

[0098] S2022, The electronic device acquires auxiliary features that affect the target search region.

[0099] The auxiliary features include at least one of the following: the predicted behavior features of the target object, and the category features of the interactive object.

[0100] The predicted behavior feature is used to characterize the predicted behavior of the target object at the next moment; the category feature of the interactive object is used to characterize the category corresponding to the interactive object.

[0101] Categories are used to distinguish different objects. For example, categories can be people, tables, walls, etc. Categories can also be person 1, person 2, table 1, table 2, etc.

[0102] For example, the target object's behavior in the previous moment was sitting, and its predicted behavior in the next moment is standing.

[0103] The category of the interactive object is used to uniquely identify the interactive object through category information.

[0104] In one possible implementation, if there is no object in the first image that shares the same category as the interactive object, the category of the interactive object can simply be category information. For example, if there is only one chair in the first image, then the category of the interactive object is "chair".

[0105] In another possible implementation, if there are other objects in the first image that belong to the same category as the interactive object, where the category of the interactive object includes both category and identifier. For example, if there are two chairs in the first image, then the category of the interactive object is Chair 1.

[0106] S2023. The electronic device splices the initial region features with the auxiliary features to obtain the second region features.

[0107] Here, different auxiliary features correspond to different splicing methods.

[0108] The second region feature is used to characterize the target search region.

[0109] S2024. The electronic device obtains the target search area by decoding the second region features through a decoder.

[0110] Compared with method 2, method 1 is simpler and more convenient to implement.

[0111] Compared to method 1, method 2 takes into account auxiliary features, so it has higher accuracy in determining the target search area.

[0112] The process by which the electronic device in S2023 fuses the initial region features with the auxiliary features to obtain the second region features will be described below.

[0113] The process may include, but is not limited to, any one of the following cases 1 to 3.

[0114] Case 1: The electronic device concatenates the initial region features with the predicted behavior features to obtain the second region features;

[0115] Scenario 2: The electronic device concatenates the initial region features with the category features of the interactive object to obtain the second region features;

[0116] Case 3: The electronic device fuses the predicted behavior features with the category features of the interactive object, and then concatenates them with the initial region features to obtain the second region features.

[0117] This application does not limit the fusion method, and it can be configured according to the actual situation. For example, the fusion method can be configured as a dot product calculation.

[0118] The process by which the electronic device obtains auxiliary features that affect the target search region in S2022 will be described below.

[0119] The process may include, but is not limited to, any one of the following implementations 1 to 3.

[0120] 1. Obtain auxiliary features when the auxiliary features include the category features of the interactive objects;

[0121] Implementation 2: When auxiliary features include predicted behavioral features, obtain auxiliary features;

[0122] 3. When the auxiliary features include predictive behavior features and category features of interactive objects, obtain the auxiliary features.

[0123] The following describes the process of obtaining auxiliary features in Implementation 1 when the auxiliary features include the category features of the interactive object.

[0124] This process may include, but is not limited to, Figure 5 S501 and S502 are shown.

[0125] S501. The electronic device uses an image segmentation algorithm to segment the first image into objects, obtaining at least two object categories.

[0126] The image segmentation algorithm can be Segmenter.

[0127] It should be noted that the first image can be segmented into objects using an image segmentation algorithm alone, resulting in at least two object categories.

[0128] Alternatively, the category information can be obtained directly at the same time as the location of each object. That is, in S2011, the categories of at least two objects can be obtained directly at the same time as the locations of at least two objects.

[0129] S502. The electronic device determines the category characteristics of the interactive object based on the category of each of the at least two objects.

[0130] Here, if the category exists in the form of a label, the label representing the category is encoded by an encoder to obtain the category features.

[0131] This process is characterized by its simple implementation logic when obtaining the category characteristics of interactive objects.

[0132] The following section explains the process of obtaining auxiliary features in implementation 2, where the auxiliary features include predicted behavioral features.

[0133] This process may include, but is not limited to, Figure 6 S601 and S604 are shown.

[0134] S601. The electronic device obtains the category characteristics and behavioral characteristics of the target object.

[0135] Since the image segmentation algorithm can obtain the categories of all objects from historical images (images including the target object before the target object was lost), S601 can be implemented as follows: the electronic device encodes the category of the target object to obtain the category feature of the target object; the electronic device identifies the behavior of the target object through a behavior detection algorithm to obtain the behavior feature of the target object.

[0136] Understandably, since the process involves the target object, the image being processed is the image that included the target object before it was lost.

[0137] S602. The electronic device concatenates the category features of the target object with the behavioral features of the target object to obtain a first feature.

[0138] Therefore, the first feature can characterize both the category and the behavior of the target object.

[0139] S603. The electronic device calls the residual network to extract and process the first image to obtain the second feature.

[0140] The second feature is used to characterize environmental information in the first image.

[0141] This application does not impose specific limitations on the residual network, and it can be configured according to actual needs. For example, the residual network can be ResNet50.

[0142] S604. The electronic device calls the second neural network to process the feature obtained by concatenating the first feature and the second feature to obtain the predicted behavior feature.

[0143] The second neural network is different from the first neural network.

[0144] The second neural network is configured as a network model for predicting behavior. An exemplary second neural network can be an MLP (Multi-Level Processing). The process of training the second neural network will not be described in detail in this application.

[0145] As can be seen, when obtaining the predicted behavior characteristics, the embodiments of this application consider not only the category of the target object, the behavior of the target object, but also environmental factors, so the accuracy of the predicted behavior is high.

[0146] To simplify the process, the influence of the second feature can be removed when obtaining the predicted behavioral features of the target behavior object. For the specific implementation process, please refer to S601 to S604, which will not be elaborated here.

[0147] The following section explains the process of obtaining auxiliary features in implementation 3, where the auxiliary features include predicted behavioral features and category features of interactive objects.

[0148] In one possible implementation, the predicted behavioral characteristics and the category characteristics of the interactive objects can be obtained through independent processes, the implementation of which can be referred to in the detailed description of implementation 1 and implementation 2.

[0149] In another possible implementation, the predicted behavioral features and the category features of the interacting objects can be obtained simultaneously through a single process. This process may include, but is not limited to, [the following]. Figure 7 S701 to S703 are shown.

[0150] S701, The electronic device acquires the category and location of the target object and the category and location of the related objects.

[0151] The relevant object is any one of the at least two objects other than the target object.

[0152] The category and location of the target object, as well as the category and location of related objects, can be obtained directly after processing historical images (images including the target object before they were lost) using an image segmentation algorithm.

[0153] S702, The electronic device calling behavior detection model detects the category and location of the target object and the category and location of the related objects to obtain the third feature.

[0154] The third feature is used to characterize the behavior between the target object and the interaction object at the current moment.

[0155] The behavior between the target object and the interaction object at the current moment can be "object 1 is sitting on chair 1".

[0156] The behavior detection model here can not only detect the current behavior of the target object, but also obtain the target object's interaction objects. Furthermore, the target object's current behavior and interaction objects are represented by a label.

[0157] Therefore, the third feature can simultaneously characterize the current behavior of the target object and the interaction objects of the target object.

[0158] S703. The electronic device calls the prediction model to perform prediction processing on the third feature to obtain the predicted behavior feature and the category feature of the interactive object.

[0159] Since the third feature can simultaneously characterize the current behavior of the target object and the objects interacting with the target object, after inputting the third feature into the behavior prediction model, the current behavior of the target object is predicted to obtain the predicted behavior, and the predicted behavior features and the category features of the interacting objects are output at the same time.

[0160] It can be seen that this process directly obtains "object 1 is sitting on chair 1" during behavior detection, which has the characteristics of being simple to implement and having good correlation.

[0161] The data processing method provided in the embodiments of this application will be described below through a complete process.

[0162] Object tracking is a common task in computer vision. A core problem is how to recover a lost target. Although many methods exist, none have completely solved this problem. The process of recovering a lost target can be broken down into two sub-problems: first, how to measure the distance to the target after loss to determine if the recovered target is the same as the previous one; this can be defined as the target feature representation problem. Second, in which region should the search area be determined after loss? A large region is computationally expensive and prone to misidentification, while a small region may result in the target not being within the area; this can be defined as the target search region problem. Regarding sub-problem 1, the target feature representation problem, a more powerful network is generally used to generate the target's corresponding feature vector. Most current work focuses on this, but it has reached an algorithmic bottleneck. Regarding sub-problem 2, the target search region problem, a common approach is to set a fixed threshold size for the search region; however, this approach lacks further research.

[0163] The solution proposed in this embodiment of the application aims to solve sub-problem 2 by adaptively generating a search area by combining environmental understanding and behavior recognition. Environmental understanding refers to detecting and recognizing the real environment in which target tracking is needed, that is, detecting the location of all objects in the environment and recognizing the attributes of each object (equivalent to the object category mentioned above). Behavior recognition refers to recognizing all behaviors of pedestrians in the current environment. Combining environmental understanding and behavior recognition essentially involves predicting pedestrian behavior and predicting the area where the behavior will occur. This area, which is the area where the pedestrian is located, can then be used as the search area for target tracking.

[0164] The process of this embodiment of the application is divided into three steps: 1) Environment understanding, that is, detecting, recognizing and segmenting scene images; 2) Behavior detection, that is, recognizing the behavior of pedestrians (between people and between people and objects), such as communicating with someone, being obscured by something, picking up something, etc.; 3) General target tracking process, except that the search area is based on the results of the first two steps.

[0165] The application scenario of this embodiment is target tracking in indoor surveillance. It consists of three algorithm models: environment understanding, behavior detection, and search region estimation. First, environment understanding employs an indoor scene instance segmentation algorithm, which identifies the category and location of all target objects in the current scene. Second, behavior detection uses a pedestrian behavior detection algorithm, which detects interactions between people and between people and objects. Finally, search region estimation, based on the results of environment understanding and behavior detection, estimates the region where pedestrians may engage in behavior in subsequent frames.

[0166] First, there is the environment understanding model. This embodiment of the application uses an instance segmentation algorithm to segment the pixels occupied by each object in the scene.

[0167] For example, such as Figure 8 As shown, before segmentation Figure 8 As shown in the image on the left, after segmentation... Figure 8 As shown in the image on the right, the segmentation algorithm identified all objects, as well as the category and location of each object.

[0168] For example, the model used for image segmentation can be Segmenter, and the algorithm framework is as follows: Figure 9 As shown, it includes: input image 901, encoder 902, mask encoder 903, and output image 904.

[0169] Secondly, there is the behavior detection model. Behavior detection in this embodiment of the application encompasses two layers of meaning: person-to-person and person-to-object. For different application scenarios, the labels of the behavior detection dataset differ from ordinary labels. For example, in car dealership surveillance, behavior labels include: entering the driver's seat, entering the right rear seat, walking past greenery, and talking to someone. Ordinary behavior detection labels are "behavior in progress," while in this embodiment of the application, they are represented as "behavior in progress + with a clearly defined interaction object." For example, "walking past greenery" indicates the behavior is walking, and the interaction object is the greenery; similarly, "entering the driver's seat" indicates the behavior is entering, and the interaction object is the driver's seat. The occurring behavior can be used to predict subsequent behaviors, and the interaction object indicates the area to be searched after the target is lost. Combining the predicted behavior with the actual behavior allows for more precise location of the search area. An example of the labels in this embodiment of the application is as follows. Figure 10 As shown, label 1 is "Object A is sitting on chair A", and label 2 is "Object B is sitting on chair B". "Sitting" is the action, and "chair" is the interactive object.

[0170] The algorithmic framework used for behavior detection is as follows: Figure 11As shown, the input image is processed by a ResNet50 to extract feature vectors, then by an MLP, and finally by fully connected layers (FC) to output the behavior classification result.

[0171] The process of obtaining Label 1 and Label 2 is explained below. This process requires understanding the model's results in the context of the environment, as described above. Figure 11 Given an input image, the environment understanding model can determine the pixels where chair A and chair B are located. The target tracking algorithm itself will detect object A and object B. The behavior detection model outputs the behaviors of object A and object B respectively, both of which are sitting. Combining the three results, we get "object A is sitting on chair A" and "object B is sitting on chair B".

[0172] Finally, there's the search region estimation model, which utilizes the results of behavior detection. The entire flowchart is as follows. Figure 12 As shown, it consists of two parts: behavior prediction and search area estimation.

[0173] Behavior prediction may include: object 1201, behavior 1202, first splicing process 1203, first encoding process 1204, feature 1 (1205), input image 1206, ResNet50 (1207), feature 2 (1208), second splicing process 1209, first MLP 1210, fully connected layer 1211, and predicted behavior 1212.

[0174] Search region estimation may include: second encoding process 1213, interactive object-category 1214, third encoding process 1215, dot product process 1216, interactive object-region 1217, fourth encoding process 1218, second MLP 1219, third concatenation process 1220, decoding process 1221, and search region 1222.

[0175] The input for behavior prediction includes object 1201, current behavior 1202, and the current frame image (input image 1206). First, the object and behavior are concatenated and encoded to obtain the corresponding feature vector (feature 1). At the same time, the current frame image is processed through ResNet50 to obtain a feature vector (feature 2). The two feature vectors are concatenated, passed through an MLP, and then through a fully connected (FC) layer to obtain the classification result of behavior prediction (predicted behavior 1212).

[0176] The search region estimation first utilizes information about the interactive object, which includes two aspects: the object's category (object 1201) and its region (interactive object - region 1217). Both are encoded as feature vectors and combined with the feature vector encoded from the behavior prediction result for decoding to obtain the final search region. The search region is image data and can be viewed as a polygonal region outside the interactive object's own region, predicted based on possible behaviors.

[0177] At this point, the search region estimation based on environmental understanding and behavior detection is complete, and this region can be directly used as the search region during target tracking. Because it combines target behavior prediction and interactive objects, it accurately predicts the region where the target may appear in the next frame, effectively solving the problem of losing track of the target during tracking.

[0178] Secondly, embodiments of this application provide a data processing apparatus, such as... Figure 13 As shown, the data processing device 130 includes a determining unit 1301 and an estimating unit 1302. Wherein:

[0179] The determining unit 1301 is configured to determine a first region in the first image when no target object is detected in the first image; the first image is an image obtained by image acquisition of the current scene; the first region is the region occupied by the interactive object of the target object in the first image;

[0180] The estimation unit 1302 is used to call the region estimation model to perform estimation processing on at least the first region to obtain the target search region, so as to perform a re-search on the target object in the target search region.

[0181] In some embodiments, the determining unit 1301 is further configured to: perform object segmentation on the first image using an image segmentation algorithm to obtain the location regions of at least two objects; identify the location region of the interactive object among the location regions of the at least two objects; and determine the location region of the interactive object as the first region.

[0182] In some embodiments, the estimation unit 1302 is further configured to: invoke a first neural network to estimate the first region features representing the first region to obtain initial region features; the first neural network is configured to estimate the initial region of the target object; obtain auxiliary features affecting the target search region; the auxiliary features include at least one of the following: the predicted behavior features of the target object, the category features of the interactive object; the predicted behavior features are used to characterize the predicted behavior of the target object at the next time step; the category features of the interactive object are used to characterize the category corresponding to the interactive object; concatenate the initial region features with the auxiliary features to obtain a second region feature; and decode the second region features through a decoder to obtain the target search region.

[0183] In some embodiments, the estimation unit 1302 is further configured to: concatenate the initial region features with the predicted behavior features to obtain the second region features; or, concatenate the initial region features with the category features of the interactive object to obtain the second region features; or, fuse the predicted behavior features with the category features of the interactive object and then concatenate them with the initial region features to obtain the second region features.

[0184] In some embodiments, where the auxiliary features include the category features of the interactive object, the estimation unit 1302 is further configured to: perform object segmentation on the first image using an image segmentation algorithm to obtain the categories of at least two objects; and determine the category features of the interactive object based on the category of each of the at least two objects.

[0185] In some embodiments, where the auxiliary features include the predicted behavior features, the estimation unit 1302 is further configured to: obtain the category features of the target object and the behavior features of the target object; concatenate the category features of the target object and the behavior features of the target object to obtain a first feature; invoke a residual network to extract and process the first image to obtain a second feature; the second feature is used to characterize environmental information in the first image; invoke a second neural network to process the feature obtained by concatenating the first feature and the second feature to obtain the predicted behavior features; the second neural network is different from the first neural network.

[0186] In some embodiments, where the auxiliary features include the category features of the interactive object and the predicted behavior features, the estimation unit 1302 is further configured to: obtain the category and location of the target object and the category and location of the related object; the related object is an object other than the target object among the at least two objects; call a behavior detection model to perform detection processing on the category and location of the target object and the category and location of the related object to obtain a third feature; the third feature is used to characterize the behavior between the target object and the interactive object at the current moment; call a prediction model to perform prediction processing on the third feature to obtain the predicted behavior features and the category features of the interactive object.

[0187] It should be noted that the data processing device provided in this application embodiment includes all the units included, which can be implemented by a processor in an electronic device; of course, it can also be implemented by specific logic circuits; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field-programmable gate array (FPGA), etc.

[0188] The descriptions of the above device embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0189] It should be noted that, in the embodiments of this application, if the above-described data processing method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0190] Thirdly, embodiments of this application provide an electronic device, including a memory and a processor. The memory stores a computer program that can run on the processor, and when the processor executes the program, it implements the steps in the data processing method provided in the above embodiments.

[0191] The following is combined with Figure 14 The electronic device 140 shown is illustrated with a structural diagram of the electronic device.

[0192] In one example, such as Figure 14 As shown, the electronic device 140 includes: a processor 1401, at least one communication bus 1402, at least one external communication interface 1403, and a memory 1404. The communication bus 1403 is configured to enable communication between these components. The external communication interface 1403 may include standard wired and wireless interfaces.

[0193] The memory 1404 is configured to store instructions and applications executable by the processor 1401, and can also cache data to be processed or already processed by the processor 1401 and various modules in the electronic device (e.g., image data, audio data, voice communication data and video communication data), which can be implemented by flash memory or random access memory (RAM).

[0194] Fourthly, embodiments of this application provide a storage medium, namely a computer-readable storage medium, on which a computer program is stored, which, when executed by a processor, implements the steps in the data processing method provided in the above embodiments.

[0195] It should be noted that the descriptions of the storage medium and device embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0196] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0197] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0198] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0199] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0200] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0201] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0202] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0203] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing method, characterized by, The method includes: If no target object is detected in the first image, a first region is determined in the first image; the first image is an image obtained by capturing images of the current scene; the first region is the area occupied by the interactive object of the target object in the first image; An initial region feature is obtained by calling a first neural network to estimate the first region feature representing the first region; the first neural network is used to estimate the initial region of the target object; auxiliary features affecting the target search region are obtained; the auxiliary features include at least one of the following: the predicted behavior feature of the target object, the category feature of the interactive object; the predicted behavior feature is used to represent the predicted behavior of the target object at the next time step; the category feature of the interactive object is used to represent the category corresponding to the interactive object; the initial region feature and the auxiliary features are concatenated to obtain the second region feature; the second region feature is decoded by a decoder to obtain the target search region, so as to re-search the target object in the target search region.

2. The method of claim 1, wherein, Determining the first region in the first image includes: The first image is segmented using an image segmentation algorithm to obtain the location regions of at least two objects; The location region of the interactive object is identified within the location regions of the at least two objects; The location area of ​​the interactive object is defined as the first area.

3. The method of claim 1, wherein, The second region feature is obtained by concatenating the initial region feature with the auxiliary feature, including: The second region feature is obtained by concatenating the initial region feature with the predicted behavior feature; Alternatively, the initial region features can be concatenated with the category features of the interactive object to obtain the second region features; Alternatively, the predicted behavior features can be fused with the category features of the interactive object, and then concatenated with the initial region features to obtain the second region features.

4. The method of claim 1, wherein, When the auxiliary features include the category features of the interactive object, obtaining the auxiliary features affecting the target search region includes: The first image is segmented using an image segmentation algorithm to obtain at least two object categories; The category characteristics of the interactive object are determined based on the category of each of the at least two objects.

5. The method of claim 1, wherein, When the auxiliary features include the predicted behavior features, obtaining the auxiliary features that influence the target search region includes: Obtain the category characteristics and behavioral characteristics of the target object; The first feature is obtained by concatenating the category features of the target object with the behavioral features of the target object; The residual network is invoked to extract and process the first image to obtain the second feature; the second feature is used to characterize the environmental information in the first image; The predicted behavior features are obtained by processing the concatenated features of the first and second features using a second neural network; the second neural network is different from the first neural network.

6. The method of claim 1, wherein, When the auxiliary features include the category features of the interactive object and the predicted behavior features, obtaining the auxiliary features affecting the target search region includes: Obtain the category and location of the target object, and the category and location of related objects; related objects are at least two objects other than the target object. A behavior detection model is invoked to detect the category and location of the target object and the category and location of the related objects to obtain a third feature; the third feature is used to characterize the behavior between the target object and the interacting object at the current moment. The prediction model is invoked to predict the third feature, thereby obtaining the predicted behavior feature and the category feature of the interactive object.

7. A data processing apparatus, characterized by, The device includes: The determining unit is configured to determine a first region in the first image when no target object is detected in the first image; the first image is an image obtained by image acquisition of the current scene; the first region is the region occupied by the interactive object of the target object in the first image; An estimation unit is configured to invoke a first neural network to estimate the first region features representing the first region to obtain initial region features; the first neural network is configured to estimate the initial region of the target object; obtain auxiliary features affecting the target search region; the auxiliary features include at least one of the following: the predicted behavior features of the target object, and the category features of the interactive object; the predicted behavior features are used to characterize the predicted behavior of the target object at the next time step; the category features of the interactive object are used to characterize the category corresponding to the interactive object; the initial region features and the auxiliary features are concatenated to obtain second region features; the second region features are decoded by a decoder to obtain the target search region, so as to re-search the target object in the target search region.

8. An electronic device comprising a memory and a processor, the memory storing a computer program executable on the processor, the processor executing the program to implement the data processing method according to any one of claims 1 to 6.

9. A storage medium having stored thereon a computer program, characterized in that When the computer program is executed by a processor, it implements the data processing method according to any one of claims 1 to 6.

Images