Data processing method and device, equipment, storage medium and computer program product

Abstract

A data processing method, device, equipment, storage medium and computer program product, the method comprising: obtaining a target image frame, the target image frame belonging to a target video; if there is a reference detection result in a target cache of the target video, extracting the reference detection result from the target cache, the reference detection result being a detection result obtained after global target detection on a reference image frame, the reference image frame being an image frame determined in the target video using a target step, the reference detection result including reference position information of a target object contained in the reference image frame, the target cache being a unit cache; performing edge expansion on an original region corresponding to the reference position information in the target image frame, and extracting an expanded region image after the edge expansion, performing regional target detection on the expanded region image to obtain a target tracking result of the target image frame; and outputting the target tracking result of the target image frame. The embodiments of the present application can improve the accuracy and efficiency of detecting target objects in a video.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, a data processing apparatus, a computer device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] With the rapid development of computer technology, the application of detecting target objects in videos is becoming increasingly widespread. Currently, target objects in each image frame of a video can be manually detected, and then further applications can be made based on the detected target objects in each image frame. For example, manually detecting cars in each image frame of a video and then recognizing the license plates, or manually detecting target objects in each image frame of a video and identifying the category of the target objects. This manual detection method is greatly affected by subjective factors, has low accuracy, and requires a lot of time, making it extremely inefficient. Summary of the Invention

[0003] This application provides a data processing method, apparatus, device, storage medium, and computer program product that can improve the accuracy and efficiency of detecting target objects in videos.

[0004] On one hand, embodiments of this application provide a data processing method, the method comprising:

[0005] Acquire a target image frame, wherein the target image frame belongs to the target video;

[0006] If a reference detection result exists in the target cache of the target video, the reference detection result is extracted from the target cache. The reference detection result is the detection result obtained after performing global target detection on a reference image frame. The reference image frame is an image frame determined in the target video using a target step size. The reference detection result includes the reference position information of the target object contained in the reference image frame. The target cache is a unit cache.

[0007] In the target image frame, the original region corresponding to the reference position information is expanded, and the expanded region image is extracted. The expanded region image is then used for target detection to obtain the target following result of the target image frame.

[0008] Output the target tracking result of the target image frame.

[0009] On one hand, embodiments of this application provide a data processing apparatus, the apparatus comprising:

[0010] An acquisition unit is used to acquire a target image frame, wherein the target image frame belongs to the target video;

[0011] The processing unit is configured to extract the reference detection result from the target cache if the target cache of the target video contains the reference detection result. The reference detection result is the detection result obtained after global target detection of the reference image frame. The reference image frame is an image frame determined in the target video using the target step size. The reference detection result includes the reference position information of the target object contained in the reference image frame. The target cache is a unit cache.

[0012] The processing unit is further configured to expand the original region corresponding to the reference position information in the target image frame, extract the expanded region image, perform regional target detection on the expanded region image, and obtain the target following result of the target image frame.

[0013] The output unit is used to output the target tracking result of the target image frame.

[0014] On one hand, embodiments of this application provide a computer device, which includes a processor, a communication interface, and a memory. The processor, the communication interface, and the memory are interconnected. The memory stores a computer program, and the processor is used to call the computer program to execute the data processing method of any of the above possible implementations.

[0015] On one hand, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements a data processing method of any possible implementation.

[0016] On one hand, embodiments of this application provide a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the data processing methods described above.

[0017] In this embodiment, a target image frame is acquired, which belongs to the target video. If a reference detection result exists in the target cache of the target video, the reference detection result is extracted from the target cache. The reference detection result is the detection result obtained after performing global target detection on the reference image frame. The reference image frame is an image frame determined in the target video using a target step size. The reference detection result includes the reference position information of the target object contained in the reference image frame, and the target cache is a unit cache. The original region corresponding to the reference position information in the target image frame is expanded, and the expanded region image is extracted. Region target detection is performed on the expanded region image to obtain the target following result of the target image frame. The target following result of the target image frame is output. In the above scheme, the terminal device automatically performs target detection, which can improve the accuracy and efficiency of detecting target objects in the video. In addition, by performing global target detection on the reference image frame to obtain the reference detection result, it is only necessary to focus on the expanded region determined according to the reference detection result for the target image frame, without needing to focus on the background region unrelated to the target object, which can further improve the accuracy and efficiency of detecting target objects in the video. Attached Figure Description

[0018] To more clearly illustrate the technical methods of the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This application provides a schematic diagram of the system architecture of a data processing system.

[0020] Figure 2 A flowchart illustrating a data processing method provided in this application embodiment. Figure 1 ;

[0021] Figure 3 This is a schematic diagram of the original area of ​​the unit involved in the embodiments of this application;

[0022] Figure 4 This is a schematic diagram of the expanded edge region image involved in an embodiment of this application;

[0023] Figure 5 This is a schematic diagram illustrating the target following results involved in an embodiment of this application;

[0024] Figure 6 A flowchart illustrating a data processing method provided in this application embodiment. Figure 2 ;

[0025] Figure 7 A flowchart illustrating a data processing method provided in this application embodiment. Figure 3 ;

[0026] Figure 8 A flowchart illustrating a data processing method provided in this application embodiment. Figure 4 ;

[0027] Figure 9 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0028] Figure 10 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0029] The technical methods in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0030] It is understood that in the specific embodiments of this application, target video, target image frame, reference image frame and other related data are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0031] This application proposes a data processing method that can improve the accuracy and efficiency of detecting target objects in videos, applicable to various fields and scenarios such as cloud technology, artificial intelligence, blockchain, connected vehicles, smart transportation, and smart homes. This data processing method can be implemented based on machine learning techniques within artificial intelligence. Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0032] In feasible embodiments, this data processing method can also be implemented based on cloud technology and / or blockchain technology. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology applied based on the cloud computing business model. It can form a resource pool, be used on demand, and is flexible and convenient. Cloud computing technology will become an important support. The back-end services of technical network systems require a large amount of computing and storage resources, such as video websites, image websites, and more portal websites. With the rapid development and application of the Internet industry, in the future, every item may have its own identification mark, which will need to be transmitted to the back-end system for logical processing. Data of different levels will be processed separately, and various industry data will all require strong system support, which can only be achieved through cloud computing. Blockchain is a string of text records (also known as blocks) that are linked and protected by cryptography. Each block contains the cryptographic hash of the previous block, the corresponding timestamp, and transaction data (usually represented by a hash value calculated using the Merkle tree algorithm). This design makes the block content difficult to tamper with. The distributed ledger connected by blockchain technology allows two parties to effectively record transactions and permanently verify these transactions. The data processing method provided in this application embodiment can specifically involve one or more of cloud technologies, including cloud storage, cloud database, and big data. For example, it retrieves the data (e.g., target video, target image frame, etc.) required to execute the data processing method from a cloud database. As another example, the data required to execute the data processing method can be stored on the blockchain in the form of blocks; the data generated by executing the data processing method (e.g., the target tracking result of the target image frame) can be stored on the blockchain in the form of blocks; furthermore, the computer device executing the data processing method can be a node device in the blockchain network.

[0033] Please see Figure 1 , Figure 1 This application provides a schematic diagram of the system architecture of a data processing system. Figure 1 The system architecture shown can be used to implement the data processing method proposed in the embodiments of this application. For example... Figure 1 As shown, the system architecture includes: terminal 11; or, the system architecture includes server 10 and terminal 11, and terminal 11 and server 10 are connected via a network.

[0034] Server 10 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. Terminal 11 can be a smartphone, tablet, laptop, desktop computer, smart voice interaction device, smart home appliance, in-vehicle terminal, etc., but is not limited to these.

[0035] The data processing method of this application can be executed by a terminal. For example, a user selects a target video, and then the terminal executes the data processing method proposed in this application on the target image frames in the target video selected by the user. Alternatively, it can be executed by a server. For example, the server executes the data processing method proposed in this application on the target image frames in the target video cached locally or in the cloud. Or, it can be executed jointly by the terminal and the server. For example, a user selects a target video, and the terminal sends the selected target video frame by frame to the server, and the server executes the data processing method proposed in this application on the target image frames (the currently sent image frames) in the target video.

[0036] In feasible embodiments, the terminal can capture target video in real time using a shooting device, and process each image frame in the real-time captured target video frame by frame using the data processing method proposed in this application to obtain the target tracking result of the target object in each image frame. The target tracking result may include the position information of the tracking box containing the target object in the image frame. Based on the position information of the tracking box, a tracking box can be generated in the image frame, and the final marked target video includes the tracking boxes containing the target object in each image frame.

[0037] In one embodiment, taking the joint execution of a terminal and a server as an example, the implementation process of this data processing method roughly includes: the server 10 obtains the target image frame sent by the terminal 11, the target image frame belonging to the target video; if a reference detection result exists in the target cache of the target video, the server 10 extracts the reference detection result from the target cache. The reference detection result is the detection result obtained after performing global target detection on the reference image frame. The reference image frame is an image frame determined in the target video using a target step size. The reference detection result includes the reference position information of the target object contained in the reference image frame, and the target cache is a unit cache; the server 10 expands the edge of the original region corresponding to the reference position information in the target image frame, extracts the expanded edge region image, performs region target detection on the expanded edge region image, and obtains the target following result of the target image frame; the target following result of the target image frame is output and fed back to the terminal 11. The above method can obtain the reference detection result by performing global target detection on the reference image frame, so that only the expanded edge region determined according to the reference detection result needs to be focused on for the target image frame, without needing to focus on the background region unrelated to the target object, which can improve the accuracy and efficiency of detecting target objects in the video.

[0038] It is understood that the system architecture diagrams described in the embodiments of this application are for the purpose of more clearly illustrating the technical methods of the embodiments of this application, and do not constitute a limitation on the technical methods provided in the embodiments of this application. As those skilled in the art will know, with the evolution of system architecture and the emergence of new business scenarios, the technical methods provided in the embodiments of this application are also applicable to similar technical problems.

[0039] The above provides a brief overview of the data processing method provided in the embodiments of this application. The specific implementation of this data processing method will be described in detail below.

[0040] Please see Figure 2 , Figure 2 A flowchart illustrating a data processing method provided in this application embodiment. Figure 1 This data processing method uses Figure 1 Taking terminal 11 as an example, the method includes the following steps S201-S203:

[0041] S201. Obtain the target image frame, which belongs to the target video.

[0042] In this embodiment, the target image frame is an image frame in the target video, and the target video is the video of the target object to be detected. The target object can be determined according to the actual application scenario. It can be an object with rigid properties (shape, structure, and appearance remain unchanged) or an object with non-rigid properties (shape, structure, and appearance have uncertainty and complexity). For example, the target object can be a water cup, an apple, or other objects with rigid properties, or a person, an animal, or other objects with non-rigid properties.

[0043] In one embodiment, the target image frame can be any image frame in the target video whose location information is not determined, or it can be an image frame captured in real time for the target video when the shooting device is shooting the target video.

[0044] S202. If a reference detection result exists in the target cache of the target video, the reference detection result is extracted from the target cache. The reference detection result is the detection result obtained after global target detection of the reference image frame. The reference image frame is an image frame determined in the target video using the target step size. The reference detection result includes the reference position information of the target object contained in the reference image frame. The target cache is a unit cache.

[0045] In the embodiments of this application, the reference image frame is an image frame determined in the target video using a target step size, which is a number of frames set based on experience. For example, the target step size can be 6 frames. When the target video is 30 FPS (Frames Per Second), the 0th frame, 6th frame, 12th frame, etc., can be determined from the target video as reference image frames at a frame rate of 6 frames (i.e., 5 FPS).

[0046] The target cache is a unit cache, which is used to store a single unit of data. Specifically, the unit cache can be a queue, stack, linked list, etc., with a length of 1.

[0047] The reference detection result can include reference location information of the target object contained in the reference image frame. This reference location information is determined based on the position information of the detection box containing the target object in the reference image frame. Specifically, the position information of the detection box containing the target object in the reference image frame can be used as the reference location information. Global object detection refers to determining the reference location information of the target objects contained in the entire image frame. This global object detection can be implemented through an object detection model. Specifically, the object detection model can be a one-stage detection model, such as the YOLO (You Only Look Once, an object recognition and localization algorithm based on deep neural networks) series, or the SSD (Single Shot MultiBox Detector) model; or it can be a multi-stage detection model, such as the R-CNN (Regions with CNN features) algorithm or MTCNN (Multi-task convolutional neural network).

[0048] In one specific implementation, the terminal can create a first thread, which can be the main thread. The terminal can determine whether a reference detection result exists in the target cache of the target video by calling the created first thread. If it is determined that a reference detection result exists in the target cache, the terminal calls the first thread to extract the reference detection result from the target cache. It should be noted that after the reference detection result is extracted from the target cache, the target cache is empty.

[0049] In one embodiment, the reference detection results existing in the target cache of the target video can be determined by a second thread, which is a sub-thread of the first thread. The second thread can perform global target detection on the reference image frame using the loaded target detection model to obtain the reference detection results and write them into the target cache. Since the target cache is a unit cache, when the target cache stores reference detection results, the new reference detection results will replace the original reference detection results stored in the target cache.

[0050] In one embodiment, the terminal can detect whether a target image frame is a reference image frame. For example, if the target step size is 6 frames and the target image frame is the 6th frame in the target video, then the target image frame can be determined to be a reference image frame in the target video. When the target image frame is determined to be a reference image frame in the target video, the terminal can use the target detection model loaded by the created second thread to perform global target detection on the target image frame. When the target image frame does not contain a target object, the terminal obtains an empty target detection result for the target image frame. In this case, it may not write the result to the target cache, or it may write the indication that the target detection result is empty to the target cache. When the target image frame contains a target object, the target detection result includes the position information of the target object in the target image frame. Specifically, this can be determined by the position information of the detection box containing the target object in the target image frame. When the target detection result contains position information, this target detection result can be written to the target cache as a new reference detection result. At this time, the target cache stores a new reference detection result, which can be provided to a certain image frame in the target video after the time point corresponding to the target image frame as prior knowledge of the target object in the target image frame. Additionally, if the target image frame is not a reference image frame in the target video, the second thread does not perform any operations on the target image frame regarding global object detection.

[0051] Understandably, consecutive image frames within a short period of time typically do not undergo significant changes. For example, when transmitting 30 image frames per second, the target object in frames 0 to 6 may only exhibit minor displacement. Therefore, when a reference detection result exists in the target buffer, this result can be used as prior knowledge of the target object in the target image frame to detect the target object in a local region of the target image frame, reducing detection time. The detailed implementation process is illustrated in the following examples.

[0052] S203. Expand the original region corresponding to the reference position information in the target image frame, extract the expanded region image, perform regional target detection on the expanded region image, and obtain the target following result of the target image frame.

[0053] The reference location information may include the location information of the detection box containing the target object in the reference image frame, specifically the coordinates of the diagonal of the detection box. In one embodiment, the original region corresponding to the reference location information in the target image frame is a rectangular region determined in the target image frame based on the coordinates of the diagonal of the detection box.

[0054] Since the original region corresponding to the reference position information in the target image frame is determined based on the reference position information of the target object in the reference image frame, and since there may be a frame number interval between the reference image frame and the target image frame (e.g., the reference image frame is the 0th frame of the target video and the target image frame is the 4th frame of the target video), the target object in the target image frame may have moved compared to the target object in the reference image frame. This would result in some of the content included by the target object not being in the original region of the target image frame. To solve this problem, this application uses edge expansion to ensure that the region containing all the content of the target object in the target image frame is obtained.

[0055] Expansion Plan 1:

[0056] In feasible embodiments, original parameters can be set for the original region. For example, if the original region is a rectangular region, the original parameter can be set to 20%, thus obtaining the expanded region with a 20% increase in width and height compared to the original region. However, this method may result in a large amount of background areas unrelated to the target object still existing in the expanded region image due to the original parameter being set too high or too low, or the expanded region image not containing all the content of the target object.

[0057] Expansion Plan Two:

[0058] To address the problem of the aforementioned edge expansion scheme one, this application employs the following method to expand the edges of the original region corresponding to the reference position information in the target image frame. In one embodiment, the terminal invokes a first thread to determine the original region corresponding to the reference position information in the target image frame, and extracts multiple unit original regions from the original region. The original region can be a rectangular region, and the number of these multiple unit original regions can be four. The shape of these unit original regions can be rectangular, and within the original region, each of the four unit original regions has one edge overlapping with another edge in the original region, and no two unit original regions will overlap the same edge of the original region. For example, Figure 3 The rectangular area containing points 11, 12, 13, and 14 is the unit original area. To determine the unit original area, an original parameter can be set. For example, if the original parameter is set to 10 pixels, then one side (height or width) of the rectangle corresponding to the unit original area can be determined by the side that overlaps with the original area. Then, the other adjacent side (width or height) of the rectangle corresponding to the unit original area can be determined using the set original parameter. Based on the height and width of the rectangle and the position of the side that overlaps with the original area, the unit original area can be determined.

[0059] Furthermore, the critical region of each of the multiple original unit regions can be determined separately. Taking the target original unit region as an example, the target original unit region is one of multiple original unit regions. The attribute parameters of the target original unit region can be obtained. Based on the attribute parameters, the expansion parameters of the target original unit region are determined. According to the expansion parameters, the critical region of the target original unit region is determined. The critical region is outside the original region in the target image frame. The critical region can be rectangular. The critical region of the target original unit region has one edge that overlaps with one edge of the original region, and this edge overlaps with one edge of the original region. For example, Figure 3 The rectangular area containing points 11, 12, 13, and 14 is the original unit region. Figure 3 The rectangular regions containing 21, 22, 23, and 24 are the critical regions of the original unit regions 11, 12, 13, and 14, respectively. The attribute parameter of the target unit's original region can be the original parameter of the target unit's original region, that is, the width or height of the rectangle corresponding to the target unit's original region. The expansion parameter of the target unit's original region can be the width or height of the rectangle corresponding to the critical region of the target unit's original region. This attribute parameter is positively correlated with the expansion parameter; that is, when the attribute parameter is large, the expansion parameter is also large. For example, Figure 3 When the height (attribute parameter) of the original region of the target unit indicated by 11 is 10 pixels, the expansion parameter of the original region of the target unit can also be 10 pixels. Then the height of the corresponding critical region 21 is 10 pixels, or... Figure 3 When the height (attribute parameter) of the original target unit region indicated by 11 is 5 pixels, the expansion parameter of the original target unit region can also be 5 pixels, then the height of the corresponding critical region 21 is 5 pixels. In one embodiment, one side (height or width) of the rectangle corresponding to the critical region of the original target unit region is determined by the side overlapping with the original target unit region, and the other side (width or height) of the rectangle corresponding to the critical region of the original target unit region can be determined by the expansion parameter, and these two sides are adjacent sides of the rectangle. For example Figure 3 The height of the rectangle corresponding to the rectangular area containing 21 is 10 pixels (expansion parameter), and Figure 3 The width of the rectangle corresponding to the rectangular area where 21 is located is determined by the overlapping sides.

[0060] In feasible embodiments, the attribute parameter can also be the area of ​​the original region of the target unit, that is, the area of ​​the critical region of the original region of the target unit can be positively correlated with the area of ​​the original region of the target unit. When the area of ​​the original region of the target unit is large, the area of ​​the critical region of the original region of the target unit is also large.

[0061] Further, if the critical region of the original target unit matches the original target unit region, then the critical region of the original target unit region is added to the region set. In one embodiment, the sum (or mean) of the pixel values ​​of the critical region of the original target unit region and the sum (or mean) of the pixel values ​​of the original target unit region can be calculated. When the difference between the sum (or mean) of the pixel values ​​of the critical region of the original target unit region and the sum (or mean) of the pixel values ​​of the original target unit region is less than a preset value, then the critical region of the original target unit region is determined to match the original target unit region. Alternatively, image features of the critical region of the original target unit region and image features of the original target unit region can be extracted. These image features can be extracted based on an image extraction model (e.g., a convolutional neural network). The similarity between the image features of the critical region of the original target unit region and the image features of the original target unit region can be calculated. When the similarity is greater than a similarity threshold, then the critical region of the original target unit region is determined to match the original target unit region. Further, the original region and the critical regions in the region set can be merged into an original expanded region, and the smallest rectangular region containing the original expanded region can be used as the expanded region. Figure 3 For example, if Figure 3 In the set of regions, the critical regions 21 and 22 can be constructed based on the critical regions 21 and 22 and the original regions as follows: Figure 4 The extended area shown.

[0062] The expanded edge region image is the image contained within the expanded edge region in the target image frame. In one embodiment, the terminal can invoke a first thread to extract the expanded edge region image, perform regional target detection on the expanded edge region image, and obtain the target tracking result of the target image frame.

[0063] Since the expanded region image contains the target object, the first thread does not need to focus on the background region unrelated to the target object. Therefore, the first thread can load a lightweight target following model to perform region target detection on the expanded region image and obtain the target following result for the target image frame. This lightweight target following model can be derived from an improved target detection model. Compared to the original target detection model, the lightweight target following model has fewer model parameters, and since it only needs to process local regions, it can reduce the amount of computation and the actual detection time, thereby improving the real-time performance and accuracy of target object detection in video. Specifically, after performing region target detection on the expanded region image, the target following model obtains a bounding box containing the target object relative to the expanded region image. Therefore, the bounding box needs to be projected back to the target image frame to obtain the target following result for the target image frame.

[0064] It should be noted that global object detection typically takes a considerable amount of time, such as 10 milliseconds, because it needs to consider a large background area and focus on target objects within a certain distance range. However, region object detection only requires a lightweight target following model, thus significantly reducing the time consumption compared to global object detection. When the second thread and the first thread are processing real-time captured target video, the reference image frame corresponding to the reference detection result stored in the target cache is usually located before the target image frame in the target video. For example, if the reference image frames are frames 0, 2, and 4 of the target video, when the second thread performs global object detection on frame 0, the first thread might already be processing frame 5. When the detection result of frame 0 is written to the target cache, the second thread is about to process frame 6, meaning the reference detection result cached in the target cache will only be used in frame 6. In summary, there is no direct relationship between the target image frame and the reference image frame corresponding to the reference detection result read from the target cache.

[0065] S204, Output the target image frame's target tracking result.

[0066] The target tracking result indicates the presence of a target object in the extended region of the target image frame. When the presence of a target object is indicated, the target tracking result includes the position information of the tracking box containing the target object in the target image frame. In one implementation, the terminal can display the tracking box of the target object in the target image frame based on the position information of the tracking box containing the target object. The user can determine whether the target object has been accurately detected by checking whether the tracking box contains the target object; that is, if the tracking box contains the target object, it is determined that the target object has been accurately detected; if the tracking box does not contain the target object, it is determined that the target object has not been accurately detected. Or, as... Figure 5 As shown, the follow box of the target object can be... Figure 5 The rectangular area indicated by 52) and the extended area ( Figure 5 The rectangular region indicated by 51 is displayed in the target image frame. Users can check the expanded region to determine if it contains the entire target object, and check the following box to determine if the target object has been accurately detected. When a user finds that the target object has not been accurately detected, they can look at the expanded region to find the cause of the problem. For example, if the expanded region does not contain the target object, it may be that the reference detection result is not applicable to the target image frame or the expanded region is inaccurate. If the expanded region contains the target object, it may be that the accuracy of the currently used following model is low, and the user can be instructed to further improve the currently used following model.

[0067] In one embodiment, the terminal can obtain a target following result sequence, which includes the target following results of each image frame in the target video. The target following results include the position information of the following box containing the target object in each image frame. The terminal can generate a following box based on the position information of the following box of the target object, determine the motion period based on the area of ​​each following box and the position of the image frame corresponding to each following box in the target video, determine the motion speed based on the motion period, and generate a motion evaluation report based on the motion speed.

[0068] In this embodiment, the motion cycle refers to the number of frames required for the target object to complete one periodic movement (such as squatting or jumping). A waveform diagram can be generated based on the area of ​​each following frame and the position of the corresponding image frame in the target video. The horizontal axis of the waveform diagram represents the position of the image frame corresponding to each following frame in the target video, specifically the time point corresponding to each image frame in the target video. The vertical axis represents the area of ​​each following frame. The motion cycle can be determined based on the waveform diagram. Taking squatting as an example, when the target object starts or ends a squat, it is standing. At this time, the area of ​​the following frame is the largest. During the squatting process, the area of ​​the following frame gradually decreases and then increases. The terminal can obtain the number of peaks from the waveform diagram, determine the number of squats (number of peaks - 1) based on the number of peaks, and then obtain the number of image frames included in the target video. Dividing the number of image frames included in the target video by the number of squats yields the number of frames required to complete one squat. Based on the number of frames required to complete one squat and the time consumed per frame, the running speed can be obtained. For example, if 300 frames are needed to complete one squat, and the target video has a frame rate of 30 FPS, then each frame takes 1 / 30 of a second. Therefore, the time required to complete one squat is 1 / 30 × 300 = 10 seconds, meaning the movement speed is once every 10 seconds. This exercise assessment report can include the obtained movement speed and the number of times the cyclical movement was completed (e.g., the number of squats). Additionally, it can include assessment results such as whether the speed is too fast or too slow, the exercise intensity, and calories burned.

[0069] In a feasible embodiment, the number of squats is determined based on the number of peaks. If the frame interval between two adjacent peaks is less than a preset number of frames, it can be considered that the periodic motion is not in progress or has not started. The waveform between the two adjacent peaks in the waveform diagram can be removed before calculating the motion speed and the number of times the periodic motion is completed.

[0070] In one embodiment, if the periodic movement is a high jump, the terminal can determine not only the number and speed of the jumps, but also the jump height. For example, when the jump begins or ends, the target object is standing. At this time, the lower boundary of the following frame is closest to the lower boundary of the target image frame. When the jump reaches its highest point, the lower boundary of the following frame is furthest from the lower boundary of the target image frame. The distance between the lower boundary of the following frame and the lower boundary of the target image frame can be determined based on the ordinate value of the diagonal of the following frame closest to its lower boundary, and the ordinate of the waveform graph can be generated. The abscissa of the waveform graph can also be generated based on the position of the image frame corresponding to each following frame in the target video. The difference between the ordinates of the troughs and the adjacent peaks in the waveform graph is determined. The jump height is obtained by dividing the actual height of the target object by the pixel value of the target object's height in the image frame (which can be determined based on the height of the rectangle corresponding to the following frame) and then multiplying by the above difference. The actual height of the target object can be obtained with the user's permission or consent.

[0071] In this embodiment, a target image frame is acquired. The target image frame belongs to the target video. If a reference detection result exists in the target cache of the target video, the reference detection result is extracted from the target cache. The reference detection result is the detection result obtained after performing global target detection on the reference image frame. The reference image frame is an image frame determined in the target video using a target step size. The reference detection result includes the reference position information of the target object contained in the reference image frame. The target cache is a unit cache. The original region corresponding to the reference position information in the target image frame is expanded, and the expanded region image is extracted. Region target detection is performed on the expanded region image to obtain the target following result of the target image frame. The target following result of the target image frame is then output. The above scheme can obtain the reference detection result by performing global target detection on the reference image frame. This allows the focus to be only on the expanded region determined by the reference detection result for the target image frame, without needing to focus on the background region unrelated to the target object. This can improve the accuracy and efficiency of detecting target objects in the video.

[0072] Please see Figure 6 , Figure 6 A flowchart illustrating a data processing method provided in this application embodiment. Figure 2 This data processing method uses Figure 1 Taking terminal 11 as an example, the method includes the following steps S601-S604:

[0073] S601. Obtain the target image frame, which belongs to the target video.

[0074] For a detailed description of S601, please refer to S201, which will not be repeated in this embodiment.

[0075] S602. If there is no reference detection result in the target cache of the target video, then obtain the target following result of the historical image frame. The historical image frame is the previous image frame of the target image frame.

[0076] It should be noted that there are two scenarios where the target cache of the target video lacks reference detection results. One is that the reference image frame does not contain the target object, resulting in an empty detection result when performing global target detection on the reference image frame. The other is that the reference detection result has already been extracted from the target cache, causing the target cache to be empty. For example, when processing a live-playing target video, if the reference image frames are frames 0, 3, and 5, and the reference detection result has already been extracted from the target cache when detecting the target object in frame 1, then when detecting the target object in frame 2, the target cache will be empty because frame 3 has not yet been processed. In this case, the target following results from historical image frames can be directly used to determine a local region image in the target image frame, thereby detecting the target object in the local region image.

[0077] In this embodiment, the reference detection results cached in the target cache are preferentially used. When no reference detection results are stored in the target cache, the target object is detected in the target image frame using the target following results of historical image frames. That is, when the target cache stores reference detection results, the cached reference detection results are used as prior information of the target object in the target image frame; when the target cache does not store reference detection results, the target following results of historical image frames are used as prior information of the target object in the target image frame. When the target object fails to be detected because it leaves the shooting interface, this method can use the cached reference detection results to obtain the detection box again, avoiding detection failure. Furthermore, by using the target following results of historical image frames or the reference detection results cached in the target cache, it is not necessary to perform global target detection on every image frame in the target video, and there is no need to pay attention to background areas unrelated to the target object, which can reduce detection time.

[0078] In one embodiment, when the target image frame is the initial image frame of the target video (i.e., the first frame of the target video), object location information in multiple sample images is obtained. This object location information refers to the location information of the target object contained in the sample images. The terminal can use an object detection model to perform global object detection on each of the multiple sample images, obtaining a detection box containing the target object in each sample image. The location information of the detection box containing the target object in each sample image is used as the object location information in each sample image. Statistical analysis is performed on each object location information to obtain average location information. This object location information can be the coordinates of the diagonal of the rectangle. The average location information can be obtained by averaging the diagonal coordinates. The average location information is used as the target tracking result of the aforementioned historical image frames. By using this embodiment, the problem of not being able to detect the target object when the initial image frame does not contain historical image frames can be solved.

[0079] In one embodiment, if the target image frame is a reference image frame in the target video, the terminal can create a second thread, which is a sub-thread of the first thread. Further, the terminal can invoke the created second thread to perform global target detection on the target image frame using a loaded target detection model (a one-stage detection model or a multi-stage detection model) to obtain the target detection result for the target image frame. This target detection result includes two cases: one where the target detection result indicates that the target image frame does not contain a target object, in which case the target detection result is empty; and another where the target detection result indicates that the target image frame contains a target object, in which case the target detection result includes the position information of the target object in the target image frame, which can be determined by the position information of the detection box containing the target object in the target image frame. When the target detection result contains position information, this target detection result can be written as a new reference detection result to the target cache. At this time, the target cache stores the new reference detection result, which can be provided to a certain image frame in the target video after the time point corresponding to the target image frame as prior knowledge of the target object in the target image frame.

[0080] S603. If the target following result of the historical image frame includes position information, then in the target image frame, the reference area corresponding to the position information included in the target following result of the historical image frame is expanded, and the expanded auxiliary area image is extracted. The auxiliary area image is then subjected to regional target detection to obtain the target following result of the target image frame.

[0081] The reference region corresponding to the position information included in the target tracking result of the historical image frame is a rectangular region determined in the target image frame based on the position information included in the target tracking result of the historical image frame.

[0082] Expansion Plan 1:

[0083] In feasible embodiments, original parameters can be set for the reference region. For example, if the reference region is a rectangular area, the original parameter can be set to 20%, thus obtaining an expanded auxiliary region with a 20% increase in width and height compared to the reference region. However, this approach may result in the auxiliary region image still containing a large amount of background areas unrelated to the target object, or the auxiliary region image not containing all the content of the target object, due to the original parameter being set too large or too small.

[0084] Expansion Plan Two:

[0085] In one embodiment, the terminal invokes a first thread to determine the reference region corresponding to the position information included in the target tracking result of the historical image frame, and extracts multiple unit reference regions from the reference region. The reference region is a rectangular area, and the number of these multiple unit reference regions can be four. The shape of each unit reference region can be rectangular, and within the reference region, each of the four unit reference regions has one edge overlapping with another edge in the reference region, and no two unit reference regions will overlap the same edge of the reference region. For example, Figure 3 The rectangular area containing points 11, 12, 13, and 14 is the unit reference area. To determine the unit reference area, an initial parameter can be set, for example, to 10 pixels. Then, the side (height or width) of the rectangle corresponding to the unit reference area can be determined by the side that overlaps with the reference area. Then, the other adjacent side (width or height) of the rectangle corresponding to the unit reference area can be determined using the set initial parameter. Based on the height and width of the rectangle and the position of the side that overlaps with the reference area, the unit reference area can be determined.

[0086] Furthermore, the critical region of each of the multiple unit reference regions can be determined separately. Taking the target unit reference region as an example, the target unit reference region is one of multiple unit reference regions. The attribute parameters of the target unit reference region can be obtained. Based on the attribute parameters, the expansion parameters of the target unit reference region are determined. According to the expansion parameters, the critical region of the target unit reference region is determined. The critical region is outside the reference region in the target image frame. The critical region can be rectangular. The critical region of the target unit reference region has one edge that overlaps with one edge of the target unit reference region, and this edge also overlaps with one edge of the reference region. For example, Figure 3 The rectangular area containing numbers 11, 12, 13, and 14 is the unit reference area. Figure 3The rectangular regions containing 21, 22, 23, and 24 are the critical regions of unit reference regions 11, 12, 13, and 14, respectively. The attribute parameters of the target unit reference region can be its original parameters, i.e., the width or height of the rectangle corresponding to the target unit reference region. The expansion parameter of the target unit reference region can be the width or height of the rectangle corresponding to the critical region of the unit reference region. This attribute parameter is positively correlated with the expansion parameter; that is, when the attribute parameter is large, the expansion parameter is also large. For example, Figure 3 When the height (attribute parameter) of the target unit reference region indicated by 11 is 10 pixels, the expansion parameter of the target unit reference region can also be 10 pixels. Then, the height of the corresponding critical region 21 is 10 pixels, or... Figure 3 When the height (attribute parameter) of the target unit reference region indicated by 11 is 5 pixels, the expansion parameter of the target unit reference region can also be 5 pixels, then the height of the corresponding critical region 21 is 5 pixels. In one embodiment, one side (height or width) of the rectangle corresponding to the critical region of the target unit reference region is determined by the side overlapping with the target unit reference region, and the other adjacent side (width or height) of the rectangle corresponding to the critical region of the target unit reference region can be determined by the expansion parameter. For example... Figure 3 The height of the rectangle corresponding to the rectangular area containing 21 is 10 pixels (expansion parameter), and Figure 3 The width of the rectangle corresponding to the rectangular area where 21 is located is determined by the overlapping sides.

[0087] In feasible embodiments, the attribute parameter can also be the area of ​​the target unit reference region, that is, the area of ​​the critical region of the target unit reference region can be positively correlated with the area of ​​the target unit reference region. When the area of ​​the target unit reference region is large, the area of ​​the critical region of the target unit reference region is also large.

[0088] Further, if the critical region of the target unit reference region matches the target unit reference region, then the critical region of the target unit reference region is added to the reference set. In one embodiment, the sum (or mean) of pixel values ​​of the critical region of the target unit reference region and the sum (or mean) of pixel values ​​of the target unit reference region can be calculated. When the difference between the sum (or mean) of pixel values ​​of the critical region of the target unit reference region and the sum (or mean) of pixel values ​​of the target unit reference region is less than a preset value, then the critical region of the target unit reference region is determined to match the target unit reference region; or, image features of the critical region of the target unit reference region and image features of the target unit reference region can be extracted. These image features can be extracted based on an image extraction model (e.g., convolutional neural network). The similarity between the image features of the critical region of the target unit reference region and the image features of the target unit reference region is calculated. When the similarity is greater than a similarity threshold, then the critical region of the target unit reference region is determined to match the target unit reference region. The critical regions in the reference region and the reference set can be merged into an original auxiliary region, and the smallest rectangular region containing the original auxiliary region is used as the expanded auxiliary region. Figure 3 For example, if Figure 3 In the region set, the critical regions 21 and 22 can be constructed based on the critical regions 21 and 22 and the reference region as follows: Figure 4 The auxiliary area shown.

[0089] The expanded auxiliary region image is the image contained within the expanded auxiliary region in the target image frame. In one embodiment, the terminal can invoke a first thread to extract the expanded auxiliary region image, perform regional target detection on the auxiliary region image, and obtain the target tracking result of the target image frame.

[0090] Since the auxiliary region image contains the target object, the first thread does not need to focus on background regions unrelated to the target object. Therefore, the first thread can load a lightweight target following model to perform region target detection on the auxiliary region image and obtain the target following result for the target image frame. This lightweight target following model can be derived from an improved target detection model. Compared to the original target detection model, the lightweight target following model has fewer model parameters, and since it only needs to process local regions, it can reduce the amount of computation and the actual detection time, thereby improving the real-time performance and accuracy of target object detection in video. Specifically, after the target following model performs region target detection on the auxiliary region image, the resulting bounding box containing the target object is relative to the auxiliary region image. Therefore, the bounding box needs to be projected back to the target image frame to obtain the target following result for the target image frame.

[0091] In this embodiment, global target detection of the reference image frame is performed by the second thread, so that the first thread does not need to focus on the background area of ​​the target image frame. At the same time, when the second thread is a background thread (a type of child thread) of the first thread, the overall time consumption depends on the first thread. By adopting a lightweight target following model, the first thread can improve the real-time performance and efficiency of detecting target objects in the video.

[0092] S604. Mark the target following results in the target image frame and output the marked target image frame.

[0093] The target tracking result indicates the presence of a target object in the auxiliary region image of the target image frame. When the presence of a target object is indicated, the target tracking result includes a tracking box containing the target object in the target image frame. In one embodiment, the target tracking result is marked in the target image frame, that is, the tracking box containing the target object is displayed in the target image frame, which makes it convenient for the user to determine whether the target object has been accurately detected by checking whether the tracking box contains the target object; or, as... Figure 5 As shown, the follow box of the target object can be... Figure 5 The rectangle indicated by 52) and the auxiliary area ( Figure 5 The rectangular box indicated by 51 is displayed in the target image frame, allowing users to easily check whether the auxiliary area contains the entire content of the target object, and to determine whether the target object has been accurately detected by checking whether the target object is contained in the following box. When the user finds that the target object has not been accurately detected, the user can check whether the detection failure is due to the auxiliary area not containing the target object.

[0094] In one embodiment, the terminal can acquire a target following result sequence, which includes the target following results for each image frame in the target video. Each target following result includes a following bounding box. Based on the area of ​​each following bounding box and the position of the corresponding image frame in the target video, a motion period is determined. Based on the motion period, a motion speed is determined, and based on the motion speed, a motion evaluation report is generated. For detailed process, please refer to step S204; this embodiment will not repeat it further.

[0095] In feasible embodiments, if when performing regional target detection on the auxiliary region image, the target image frame indicates that there is no target object, it can be determined that the target image frame does not include the target object to be detected.

[0096] In this embodiment, a target image frame is obtained, which belongs to the target video. If there is no reference detection result in the target cache of the target video, the target following result of a historical image frame is obtained. The historical image frame is the previous image frame of the target image frame. If the target following result of the historical image frame includes position information, the reference region corresponding to the position information included in the target following result of the historical image frame is expanded in the target image frame, and the expanded auxiliary region image is extracted. Region target detection is performed on the auxiliary region image to obtain the target following result of the target image frame. The target following result is marked in the target image frame, and the marked target image frame is output. Through the above scheme, the target following result of the historical image frame can be used to make it possible to focus only on the auxiliary region image for the target image frame, without having to focus on the background region unrelated to the target object, which can improve the accuracy and efficiency of detecting target objects in the video.

[0097] In one embodiment, the processing logic of the data processing method can be encapsulated as a Software Development Kit (SDK), which is a collection of development tools used to build application software. The SDK is embedded in a host application, which can send target image frames from the target video to the SDK in real time via the application programming interface (API) provided by the SDK. This allows the SDK to execute the data processing method proposed in this application to obtain the target image frame tracking result. See also... Figure 7 , Figure 7 A flowchart illustrating a data processing method provided in this application embodiment. Figure 3 The method includes the following steps S701-S705:

[0098] S701 and SDK receive target image frames from the target video sent by the host application through API calls.

[0099] S702, the SDK uses a first thread to check if a reference detection result exists in the target cache of the target video. This target cache can be a queue of length 1 created by the SDK using a second thread. The reference detection result cached in the target cache can be obtained by the second thread performing global target detection on the reference image frame using the loaded target detection model.

[0100] S703 and SDK utilize the first thread to detect the existence of reference detection results in the target cache of the target video. Then, they extract the reference detection results from the target cache. The reference detection results include the reference position information of the target object contained in the reference image frame. The first thread expands the original region corresponding to the reference position information in the target image frame, extracts the expanded region image, performs region target detection on the expanded region image, obtains the target following result of the target image frame, and outputs the target following result of the target image frame to the host application.

[0101] S704 and SDK use the first thread to detect that the target cache of the target video is empty, then obtain the target following result of the historical image frame. The historical image frame is the previous image frame of the target image frame. When it is determined that the target following result of the historical image frame includes position information, the reference area corresponding to the position information included in the target following result of the historical image frame is expanded in the target image frame, and the expanded auxiliary area image is extracted. The auxiliary area image is then used for regional target detection to obtain the target following result of the target image frame, and the target following result of the target image frame is output to the host application.

[0102] S705: The host application receives the target tracking result of the target image frame sent by the SDK and executes downstream tasks based on the target tracking result of the target image frame.

[0103] The downstream tasks performed by the host application can be determined based on the application scenario. For example, the application scenario may include performing actions such as lengthening legs, adding special effects, changing clothes, and beautifying / slimming on target objects in a target video.

[0104] In this embodiment, by encapsulating the data processing method proposed in this application into an SDK, the solution of this application can be plug-and-play. This allows the host application to simply load the SDK beforehand and then send the target image frame from the target video to the SDK to start the SDK to execute the data processing method proposed in this application, thereby improving the ease of use of this solution.

[0105] It should be noted that when processing real-time captured target video, the second thread runs in parallel with the first thread. When the SDK receives the target image frame, and the target image frame is a reference image frame, the first thread executes S703 or S704 to determine the target tracking result of the target image frame. At the same time, the second thread performs global target detection on the target image frame. Since the second thread needs to consider a large number of background areas in the reference image frame that are unrelated to the target object, a target detection model is used to ensure that the position information of the target object in the reference image frame (i.e., the target detection result) can be accurately obtained. Taking the YOLO model as an example, the YOLO model can accurately identify target objects within 5 meters of the shooting device for image frames with a resolution of 320×320, with a processing time of 11 milliseconds per frame. The first thread, not needing to focus on a large background area, can use a lightweight target-following model. For example, a lightweight target-following model could be an improved YOLO model (the improvement could be a reduction in the number of model layers, etc.). The obtained expanded region can be scaled to 96×96, and then input into the improved YOLO model for region target detection. Based on the first expansion scheme, the time per frame for region target detection is approximately 0.6 milliseconds. Based on the second expansion scheme, the time per frame for region target detection is only slightly greater than 0.6 milliseconds. The second thread is a sub-thread of the first thread. When this sub-thread is a background thread (background threads will close when the main thread closes, even if they are not finished executing), the overall latency of the SDK executing this scheme depends on the first thread. It can be guaranteed that when a target object exists in the target image frame, the time is approximately 0.6 milliseconds, and when no target object exists, the time is 0 seconds (because there is no need to call the lightweight target-following model). Therefore, the overall latency is less than 1 millisecond, which is negligible. Therefore, the process from the SDK receiving the target image frame sent by the host application to the SDK returning the target image frame's target tracking result to the host application has near-zero latency (less than 1 millisecond per frame), exhibiting high real-time performance. Simultaneously, when the first thread detects target objects in the target image frame based on the target tracking results of historical image frames, even if the target object cannot be detected due to moving out of the shooting interface, the target object's position information can be re-determined using the reference detection result determined by the second thread. This avoids detection failures and improves the accuracy of target object detection in the target video.

[0106] Please see Figure 8 , Figure 8 A flowchart illustrating a data processing method provided in this application embodiment. Figure 4 The method includes:

[0107] The terminal (or SDK) uses a first thread to receive target image frames from the target video input at a frame rate of 30 FPS and performs regional target detection on the target image frames. It also uses a second thread to receive reference image frames from the target video input at a frame rate of 5 FPS and performs global target detection on the reference image frames.

[0108] In one embodiment, the terminal can use the target detection model loaded by the second thread to perform global target detection on the reference image frame, obtain the reference detection result, and write the reference detection result into the target cache (a queue of length 1).

[0109] In one embodiment, the terminal can use the first thread to access the target cache in the second thread to determine whether a reference detection result exists in the target cache. Since global target detection takes longer than regional target detection, the reference image frame processed by the second thread is often before the target image frame processed by the first thread. For example, the second thread is still processing frame 0 (reference image frame) in the target video, while the first thread is already processing frame 4 (target image frame) in the target video.

[0110] Furthermore, when the terminal determines that a reference detection result exists in the target cache using the first thread, it can expand the edges of the original region corresponding to the reference position information in the target image frame based on the reference position information included in the reference detection result, and extract the expanded edge region image (i.e., Figure 8 The target image (a local region image) is processed using a target following model loaded in the first thread to perform target detection on the expanded region image, resulting in the target following result for the target image frame, which is then output. Specifically, after performing target detection on the expanded region image, the target following model obtains a bounding box containing the target object relative to the expanded region image. Therefore, the bounding box needs to be projected back onto the target image frame to obtain the target following result for the target image frame.

[0111] To ensure optimal real-time performance, an expansion method can be used to expand the original region corresponding to the reference position information within the target image frame. For example, the expanded region increases by 20% compared to the original region. During training, the target following model needs to fully consider various possibilities of rapid movement of the target object, ensuring that even in extreme cases where parts of the target object in the target image frame are not within the expanded region, the model can accurately estimate the target object's position. For instance, if the expanded region does not include the target object's left hand, the target following model can predict its approximate location and generate a rectangular region that partially exists outside the expanded region but includes the left hand.

[0112] In addition, the target following model can also output the category of the target object (e.g., human, cat, dog) and / or the target object's movement speed, etc.

[0113] In another embodiment, when the terminal determines that no reference detection result exists in the target cache using the first thread, it can expand the edges of the reference region corresponding to the position information included in the target following result of the historical image frame in the target image frame based on the position information included in the target following result of the historical image frame, and extract the expanded auxiliary region image (i.e. Figure 8 The system uses a local region image (in the image above) and a target following model loaded in the first thread to perform region target detection on the auxiliary region image, obtaining the target following result for the target image frame, and outputting the target following result for the target image frame. Additionally, if the target following result for a historical image frame does not include position information, it can be determined that the target following result for the target image frame is empty. Specifically, after the target following model performs region target detection on the auxiliary region image, the resulting bounding box containing the target object is relative to the auxiliary region image; therefore, the bounding box needs to be projected back to the target image frame to obtain the target following result for the target image frame.

[0114] Accordingly, to ensure optimal real-time performance, the aforementioned edge expansion method can be used to expand the reference region corresponding to the position information of the target tracking results in the historical image frames within the target image frame. Similarly, the target tracking model needs to fully consider various possibilities of rapid movement of the target object during training, so that even in extreme cases where part of the target object in the target image frame is not within the reference region, the target tracking model can accurately estimate the position information of the target object during actual detection.

[0115] Target detection models that can accurately detect objects often consume a lot of time due to computational complexity or large model size, resulting in poor real-time performance. Furthermore, the enormous computing power required by target detection models contradicts the low power consumption of mobile devices. This embodiment allows the first and second threads to collaborate. The second thread, which does not affect the overall processing time, performs global target detection, while the first thread uses an ultra-lightweight target following model to perform region target detection. This enables near-zero latency on terminal devices while improving the accuracy of target object detection.

[0116] The methods of the embodiments of this application have been described in detail above. To facilitate better implementation of the methods of the embodiments of this application, the apparatus of the embodiments of this application is provided below. Please refer to... Figure 9 , Figure 9 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. The data processing device 90 may include:

[0117] Acquisition unit 901 is used to acquire target image frames, which belong to the target video;

[0118] The processing unit 902 is used to extract the reference detection result from the target cache if the target cache of the target video contains the reference detection result. The reference detection result is the detection result obtained after global target detection of the reference image frame. The reference image frame is an image frame determined in the target video using the target step size. The reference detection result includes the reference position information of the target object contained in the reference image frame. The target cache is a unit cache.

[0119] The aforementioned processing unit 902 is further configured to expand the original region corresponding to the reference position information in the target image frame, extract the expanded region image, perform regional target detection on the expanded region image, and obtain the target following result of the target image frame.

[0120] Output unit 903 is used to output the target tracking result of the target image frame.

[0121] In one embodiment, the processing unit 902 is specifically used to: determine the original region corresponding to the reference position information in the target image frame; extract multiple unit original regions from the original region; determine the critical region of each unit original region; if the critical region of the target unit original region matches the target unit original region, add the critical region of the target unit original region to the region set, and the target unit original region is one of the multiple unit original regions; merge the original region and the critical region in the region set into the expanded edge region.

[0122] In one embodiment, the acquisition unit 901 is specifically used to: acquire attribute parameters of the original region of the target unit;

[0123] The aforementioned processing unit 902 is specifically used to: determine the expansion parameters of the original region of the target unit based on the attribute parameters of the original region of the target unit, wherein the attribute parameters are positively correlated with the expansion parameters; and determine the critical region of the original region of the target unit according to the expansion parameters.

[0124] In one embodiment, the processing unit 902 is specifically used to: if the target image frame is a reference image frame in the target video, perform global target detection on the target image frame to obtain the target detection result of the target image frame; if the target detection result contains location information, write the target detection result as a new reference detection result into the target cache.

[0125] In one embodiment, the target tracking result of the target image frame is determined in a first thread, and the reference detection result is determined in a second thread, which is a sub-thread of the first thread.

[0126] In one embodiment, the acquisition unit 901 is specifically used to: if there is no reference detection result in the target cache of the target video, acquire the target following result of the historical image frame, wherein the historical image frame is the previous image frame of the target image frame;

[0127] The aforementioned processing unit 902 is specifically used for: if the target following result of the historical image frame includes position information, then in the target image frame, the reference region corresponding to the position information included in the target following result of the historical image frame is expanded, and the expanded auxiliary region image is extracted, and the auxiliary region image is subjected to regional target detection to obtain the target following result of the target image frame.

[0128] The output unit 903 is specifically used to: mark the target following result in the target image frame and output the marked target image frame.

[0129] In one embodiment, when the target image frame is the initial image frame of the target video, the acquisition unit 901 is specifically used to: acquire object location information in multiple sample images;

[0130] The aforementioned processing unit 902 is specifically used to: perform statistical analysis on the position information of each object to obtain average position information; and use the average position information as the target tracking result of historical image frames.

[0131] In one embodiment, the acquisition unit 901 is specifically used to: acquire a target following result sequence, the target following result sequence including the target following results of each image frame in the target video;

[0132] The aforementioned processing unit 902 is specifically used to: generate tracking boxes based on the target tracking results of each image frame; determine the motion period based on the area of ​​each tracking box and the position of the corresponding image frame in the target video; determine the motion speed based on the motion period; and generate a motion evaluation report based on the motion speed.

[0133] It is understood that the functions of each functional unit of the data processing device described in the embodiments of this application can be specifically implemented according to the methods in the above method embodiments, and the specific implementation process can be referred to the relevant descriptions in the above method embodiments, which will not be repeated here.

[0134] In this embodiment, a target image frame is acquired. The target image frame belongs to the target video. If a reference detection result exists in the target cache of the target video, the reference detection result is extracted from the target cache. The reference detection result is the detection result obtained after global target detection of the reference image frame. The reference image frame is an image frame determined in the target video using the target step size. The reference detection result includes the reference position information of the target object contained in the reference image frame. The target cache is a unit cache. The original region corresponding to the reference position information in the target image frame is expanded, and the expanded region image is extracted. Region target detection is performed on the expanded region image to obtain the target following result of the target image frame. The target following result of the target image frame is output, which can improve the accuracy and efficiency of detecting target objects in the video.

[0135] like Figure 10 As shown, Figure 10 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. The internal structure of the computer device 100 is as follows: Figure 10 As shown, it includes: one or more processors 1001, memory 1002, and communication interface 1003. The processors 1001, memory 1002, and communication interface 1003 can be connected via bus 1004 or other means. This embodiment of the application takes the connection via bus 1004 as an example.

[0136] The processor 1001 (or CPU, Central Processing Unit) is the computing and control core of the computer device 100. It can parse various instructions within the computer device 100 and process various data. For example, the CPU can parse power-on / off commands sent by the user to the computer device 100 and control the computer device 100 to perform power-on / off operations; it can also transmit various interactive data between internal structures of the computer device 100, and so on. The communication interface 1003 may optionally include standard wired interfaces or wireless interfaces (such as Wi-Fi, mobile communication interfaces, etc.), and is controlled by the processor 1001 for sending and receiving data. The memory 1002 is the storage device in the computer device 100, used to store computer programs and data. It is understood that the memory 1002 here can include both the computer device 100's built-in memory and extended memory supported by the computer device 100. The memory 1002 provides storage space for storing the operating system of the computer device 100, which may include, but is not limited to, Windows, Linux, Android, iOS, etc., and this application does not limit this to any particular system. In one embodiment, the processor 1001 performs the following operations by running a computer program stored in the memory 1002:

[0137] Acquire the target image frame, which belongs to the target video;

[0138] If a reference detection result exists in the target cache of the target video, the reference detection result is extracted from the target cache. The reference detection result is the detection result obtained after global target detection of the reference image frame. The reference image frame is the image frame determined in the target video using the target step size. The reference detection result includes the reference position information of the target object contained in the reference image frame. The target cache is a unit cache.

[0139] In the target image frame, the original region corresponding to the reference position information is expanded, and the expanded region image is extracted. The expanded region image is then used for target detection to obtain the target following result of the target image frame.

[0140] Output the target image frame and the target tracking result.

[0141] In one embodiment, the processor 1001 is specifically configured to: determine the original region corresponding to the reference position information in the target image frame; extract multiple unit original regions from the original region; determine the critical region of each unit original region; if the critical region of the target unit original region matches the target unit original region, add the critical region of the target unit original region to the region set, wherein the target unit original region is one of the multiple unit original regions; and merge the original region and the critical region in the region set into an expanded region.

[0142] In one embodiment, the processor 1001 is specifically used to: obtain attribute parameters of the original region of the target unit; determine the expansion parameters of the original region of the target unit based on the attribute parameters of the original region of the target unit, wherein the attribute parameters are positively correlated with the expansion parameters; and determine the critical region of the original region of the target unit according to the expansion parameters.

[0143] In one embodiment, the processor 1001 is specifically configured to: if the target image frame is a reference image frame in the target video, perform global target detection on the target image frame to obtain the target detection result of the target image frame; if the target detection result contains location information, write the target detection result as a new reference detection result into the target cache.

[0144] In one embodiment, the target tracking result of the target image frame is determined in a first thread, and the reference detection result is determined in a second thread, which is a sub-thread of the first thread.

[0145] In one embodiment, the processor 1001 is specifically configured to: if there is no reference detection result in the target cache of the target video, obtain the target following result of a historical image frame, wherein the historical image frame is the previous image frame of the target image frame; if the target following result of the historical image frame includes position information, expand the reference region corresponding to the position information included in the target following result of the historical image frame in the target image frame, extract the expanded auxiliary region image, perform regional target detection on the auxiliary region image, and obtain the target following result of the target image frame; mark the target following result in the target image frame, and output the marked target image frame.

[0146] In one embodiment, the processor 1001 is specifically used to: acquire object location information in multiple sample images; perform statistical analysis on each object location information to obtain average location information; and use the average location information as the target tracking result of historical image frames.

[0147] In one embodiment, the processor 1001 is specifically used to: acquire a target following result sequence, the target following result sequence including the target following result of each image frame in the target video; generate a following box according to the target following result of each image frame; determine the motion period according to the area of ​​each following box and the position of the image frame corresponding to each following box in the target video; determine the motion speed according to the motion period; and generate a motion evaluation report according to the motion speed.

[0148] In specific implementations, the processor 1001, memory 1002, and communication interface 1003 described in the embodiments of this application can execute the implementation method described in the data processing method provided in the embodiments of this application, or the implementation method described in the data processing device provided in the embodiments of this application, which will not be repeated here.

[0149] In this embodiment, a target image frame is acquired. The target image frame belongs to the target video. If a reference detection result exists in the target cache of the target video, the reference detection result is extracted from the target cache. The reference detection result is the detection result obtained after global target detection of the reference image frame. The reference image frame is an image frame determined in the target video using the target step size. The reference detection result includes the reference position information of the target object contained in the reference image frame. The target cache is a unit cache. The original region corresponding to the reference position information in the target image frame is expanded, and the expanded region image is extracted. Region target detection is performed on the expanded region image to obtain the target following result of the target image frame. The target following result of the target image frame is output, which can improve the accuracy and efficiency of detecting target objects in the video.

[0150] This application also provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the data processing method described in any of the possible implementations above. Specific implementations are described above and will not be repeated here.

[0151] This application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the data processing methods described above. Specific implementations can be found in the foregoing description and will not be repeated here.

[0152] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0153] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0154] The above-disclosed embodiments are only some of the embodiments of this application, and should not be construed as limiting the scope of this application. Therefore, any equivalent changes made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A data processing method, characterized in that, The method includes: Acquire a target image frame, wherein the target image frame belongs to the target video; If a reference detection result exists in the target cache of the target video, the reference detection result is extracted from the target cache. The reference detection result is the detection result obtained after performing global target detection on a reference image frame. The reference image frame is an image frame determined in the target video using a target step size. The reference detection result includes the reference position information of the target object contained in the reference image frame. The target cache is a unit cache. Determine the original region corresponding to the reference position information in the target image frame, and extract multiple unit original regions from the original region; The critical region of each unit original region is determined separately; wherein, the critical region is outside the original region in the target image frame, and the critical region of the target unit original region has an edge that overlaps with an edge of the target unit original region, and this edge overlaps with an edge of the original region. If the critical region of the target unit's original region matches the target unit's original region, then the critical region of the target unit's original region is added to the region set, where the target unit's original region is one of the multiple unit original regions; The original region and the critical region in the region set are merged into an expanded region. Extract the expanded region image, perform regional target detection on the expanded region image, and obtain the target tracking result of the target image frame; Output the target tracking result of the target image frame.

2. The method according to claim 1, characterized in that, Determining the critical region of the original region of the target unit includes: Obtain the attribute parameters of the original region of the target unit; Based on the attribute parameters of the original region of the target unit, the expansion parameters of the original region of the target unit are determined, wherein the attribute parameters are positively correlated with the expansion parameters; Based on the expansion parameters, determine the critical region of the original region of the target unit.

3. The method according to claim 1, characterized in that, The method further includes: If the target image frame is a reference image frame in the target video, then global target detection is performed on the target image frame to obtain the target detection result of the target image frame; If the target detection result contains location information, then the target detection result is written into the target cache as a new reference detection result.

4. The method according to any one of claims 1-3, characterized in that, The target tracking result of the target image frame is determined in a first thread, and the reference detection result is determined in a second thread, which is a sub-thread of the first thread.

5. The method according to claim 1, characterized in that, The method further includes: If the reference detection result is not present in the target cache of the target video, then the target following result of the historical image frame is obtained, where the historical image frame is the previous image frame of the target image frame. If the target following result of the historical image frame includes position information, then in the target image frame, the reference region corresponding to the position information included in the target following result of the historical image frame is expanded, and the expanded auxiliary region image is extracted. The auxiliary region image is then subjected to regional target detection to obtain the target following result of the target image frame. The target following result is marked in the target image frame, and the marked target image frame is output.

6. The method according to claim 5, characterized in that, When the target image frame is the initial image frame of the target video, the step of obtaining the target following result of the historical image frames includes: Obtain object location information from multiple sample images; Statistical analysis of the location information of each object is performed to obtain the average location information; The average position information is used as the target following result of the historical image frame.

7. The method according to claim 1, characterized in that, The method further includes: Obtain a target following result sequence, the target following result sequence including the target following results of each image frame in the target video; Based on the target tracking results of each image frame, a tracking box is generated. Based on the area of ​​each tracking box and the position of the corresponding image frame in the target video, the motion period is determined. The movement speed is determined based on the movement cycle, and a movement assessment report is generated based on the movement speed.

8. A data processing apparatus, characterized in that, The device includes: An acquisition unit is used to acquire a target image frame, wherein the target image frame belongs to the target video; The processing unit is configured to extract the reference detection result from the target cache if the target cache of the target video contains the reference detection result. The reference detection result is the detection result obtained after global target detection of the reference image frame. The reference image frame is an image frame determined in the target video using the target step size. The reference detection result includes the reference position information of the target object contained in the reference image frame. The target cache is a unit cache. The processing unit is further configured to expand the original region corresponding to the reference position information in the target image frame, extract the expanded region image, perform regional target detection on the expanded region image, and obtain the target following result of the target image frame. The output unit is used to output the target tracking result of the target image frame; The processing unit is configured to: determine the original region corresponding to the reference position information in the target image frame; extract multiple unit original regions from the original region; determine the critical region of each unit original region; if the critical region of the target unit original region matches the target unit original region, add the critical region of the target unit original region to the region set, wherein the target unit original region is one of the multiple unit original regions; merge the original region and the critical region in the region set into an expanded region; wherein the critical region is outside the original region in the target image frame, and the critical region of the target unit original region has an edge that overlaps with an edge of the target unit original region, and this edge overlaps with an edge of the original region.

9. A computer device, characterized in that, The computer device includes a memory, a communication interface, and a processor, wherein the memory, the communication interface, and the processor are interconnected; the memory stores a computer program, and the processor calls the computer program stored in the memory to implement the data processing method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the data processing method according to any one of claims 1-7.

11. A computer program product, characterized in that, The computer program product includes a computer program or computer instructions, which, when executed by a processor, implement the data processing method according to any one of claims 1-7.

Images