Video processing method and device, computer device and storage medium

Abstract

The application relates to a video processing method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: receiving a to-be-processed video uploaded by a client; performing coordinate conversion on the to-be-processed video, and sending a first frame of the to-be-processed image after the coordinate conversion to the client; receiving a labeling result of the first frame of the to-be-processed image uploaded by the client, identifying and tracking a target object in the to-be-processed video according to the labeling result, generating a target video, and sending the target video to the client. The method can produce a target video for identifying and tracking a target object.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer image, in particular to a video processing method and device, computer equipment, storage medium and computer program product. BACKGROUND

[0002] With the development of computer vision, great convenience is provided for people's life, and it has become an integral part of various intelligent / autonomous systems in manufacturing, inspection, document analysis, medical diagnosis, and military fields.

[0003] In the related art, only the target object in a video can be identified, and the tracking of the target object in the video cannot be realized. SUMMARY

[0004] Therefore, it is necessary to provide a video processing method and device, computer equipment, computer readable storage medium and computer program product capable of tracking and identifying target objects.

[0005] In a first aspect, the present application provides a video processing method applied to a server, the method comprising:

[0006] receiving a to-be-processed video uploaded by a client;

[0007] performing coordinate conversion on the to-be-processed video, and sending a first frame of the to-be-processed image after coordinate conversion to the client;

[0008] receiving a labeling result of the first frame of the to-be-processed image uploaded by the client, identifying and tracking a target object in the to-be-processed video according to the labeling result and generating a target video, and sending the target video to the client.

[0009] In one of the embodiments, the identifying and tracking the target object in the to-be-processed video according to the labeling result and generating the target video comprises:

[0010] when the labeling result is qualified, identifying and tracking the target object in the to-be-processed video after coordinate conversion and generating the target video;

[0011] when the labeling result is unqualified, performing coordinate conversion on the to-be-processed video according to the coordinate information carried in the labeling result, and identifying and tracking the target object in the to-be-processed video after coordinate conversion and generating the target video.

[0012] In one of the embodiments, the identifying and tracking the target object in the to-be-processed video after coordinate conversion and generating the target video comprises:

[0013] identify a target object in the to-be-processed video, to obtain each frame of action recognition image in the to-be-processed video;

[0014] track the target object in the to-be-processed video, to obtain each frame of target tracking image in the to-be-processed video;

[0015] match each frame of action recognition image and each frame of target tracking image, to obtain each frame of target image;

[0016] connect each frame of target image, to obtain a target video.

[0017] In one of the embodiments, after the matching of each frame of action recognition image and each frame of target tracking image to obtain each frame of target image, the method further comprises:

[0018] calculate a feature value of each target object in the target image;

[0019] for each target object in the target image, classify the target object according to the feature value of the target object, to obtain a category corresponding to the target object.

[0020] In a second aspect, the application further provides a video processing method applied to a client, which comprises:

[0021] sending a to-be-processed video to a server;

[0022] receiving a first frame of to-be-processed image after coordinate conversion sent by the server, judging the first frame of to-be-processed image after coordinate conversion, generating a labeling result, and sending the labeling result to the server; the first frame of to-be-processed image is obtained by the server through coordinate conversion on the to-be-processed video;

[0023] receiving a target video sent by the server, the target video being generated by the server through identification and tracking of a target object in the to-be-processed video according to the labeling result.

[0024] In a third aspect, the application further provides a video processing device applied to a server, which comprises:

[0025] a receiving module, configured to receive a to-be-processed video uploaded by a client;

[0026] a coordinate conversion module, configured to perform coordinate conversion on the to-be-processed video, and send a first frame of to-be-processed image after coordinate conversion to the client;

[0027] an identification module, configured to receive a labeling result of the first frame of to-be-processed image uploaded by the client, identify and track a target object in the to-be-processed video according to the labeling result, generate a target video, and send the target video to the client.

[0028] In a fourth aspect, the present application provides a video processing device applied to a client, the device comprising:

[0029] a sending module configured to send a to-be-processed video to a server;

[0030] a labeling receiving module configured to receive a first to-be-processed image after coordinate conversion sent by the server, and to judge the first to-be-processed image after coordinate conversion, generate a labeling result, and send the labeling result to the server; the first to-be-processed image is obtained by the server through coordinate conversion on the to-be-processed video;

[0031] a target video receiving module configured to receive a target video sent by the server, the target video being generated by the server through identification and tracking on a target object in the to-be-processed video according to the labeling result.

[0032] In a fifth aspect, the present application provides a computer device. The computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the steps of the method in any one of the above embodiments when executing the computer program.

[0033] In a fourth aspect, the present application provides a computer readable storage medium. The computer readable storage medium stores a computer program, and the computer program implements the steps of the method in any one of the above embodiments when executed by a processor.

[0034] In a fifth aspect, the present application provides a computer program product. The computer program product comprises a computer program, and the computer program implements the steps of the method in any one of the above embodiments when executed by a processor.

[0035] The above video identification method, device, computer device, storage medium and computer program product, the server first receives a to-be-processed video uploaded by the client, then performs coordinate conversion on the to-be-processed video, and sends a first to-be-processed image after coordinate conversion to the client, the client judges the coordinate conversion effect of the first to-be-processed image, and generates a labeling result, which can ensure the quality of coordinate conversion of the to-be-processed video, so as to improve the accuracy of subsequent target video generation. Then, the server tracks and identifies a target object in the to-be-processed image according to the labeling of the first to-be-processed image uploaded by the client, and generates a target video capable of identifying and tracking the target object at the same time. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 It is an application environment diagram of the video processing method in one embodiment;

[0037] Figure 2 It is a flowchart of the video processing method in one embodiment;

[0038] Figure 3 Fig. 1 is a diagram illustrating a football field coordinate conversion in one embodiment;

[0039] Figure 4 Fig. 2 is a diagram illustrating a flow of a video processing method in another embodiment;

[0040] Figure 5 Fig. 3 is a diagram illustrating a frame in a target video in one embodiment;

[0041] Figure 6 Fig. 4 is a diagram illustrating a GUI running in one embodiment;

[0042] Figure 7 Fig. 5 is a diagram illustrating a GUI interface in one embodiment;

[0043] Figure 8 Fig. 6 is a diagram illustrating a GUI interface in another embodiment;

[0044] Figure 9 Fig. 7 is a diagram illustrating a QT transmission in one embodiment;

[0045] Figure 10 Fig. 8 is a block diagram illustrating a video processing apparatus in one embodiment;

[0046] Figure 11 Fig. 9 is a block diagram illustrating a video processing apparatus in another embodiment;

[0047] Figure 12 Fig. 10 is a diagram illustrating an internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0048] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0049] The video processing method provided by the embodiments of the present application can be applied to, for example, Figure 1The application environment shown. Among them, the client 102 communicates with the server 104 through the network. The data storage system can store the data required by the server 104 to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers. The server 104 first receives the to-be-processed video uploaded by the client 102, then performs coordinate conversion on the to-be-processed video to obtain the coordinate-converted to-be-processed video, and sends the first frame of the coordinate-converted to-be-processed image in the coordinate-converted to-be-processed video to the client 102. The client 102 judges the coordinate conversion effect according to the first frame of the to-be-processed image, generates a labeling result, and sends the labeling result to the server 104. The server 104 tracks and identifies the target object in the to-be-processed video according to the labeling result and generates a target video. Finally, the target video is sent to the client 102, wherein the target video can realize identification and tracking of the target object at the same time. Among them, the client 102 can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things device can be a smart speaker, a smart TV, a smart air conditioner, a smart vehicle device, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The server 104 can be implemented by an independent server or a server cluster composed of multiple servers.

[0050] In one embodiment, as shown in Figure 2 A video processing method is provided, which is applied to the server 104 in Figure 1 for example, including the following steps:

[0051] S202, receiving a to-be-processed video uploaded by a client.

[0052] Among them, the to-be-processed video refers to the video shot by a camera, a mobile phone or the like. It can be a video shot in any scene, such as a football video, a basketball video, etc., which is not limited here.

[0053] S204, performing coordinate conversion on the to-be-processed video, and sending the first frame of the coordinate-converted to-be-processed image to the client.

[0054] Among them, the first frame of the to-be-processed image refers to the first frame of the image in the coordinate-converted to-be-processed video after the to-be-processed video is coordinate-converted.

[0055] Specifically, after the server receives the to-be-processed video uploaded by the client, it performs coordinate conversion on the to-be-processed video, and then sends the first frame of the to-be-processed image in the coordinate-converted video to the client, so that the client judges the quality of coordinate conversion through the first frame of the to-be-processed image.

[0056] In one embodiment, the coordinate conversion of the to-be-processed video can be performed by a pre-trained coordinate conversion model, such as a coordinate conversion neural network, to obtain the coordinate-converted to-be-processed video.

[0057] In one embodiment, in particular in combination with Figure 3 as shown, Figure 3 is a football field coordinate conversion schematic diagram in one embodiment. The left side is a certain frame of image in a photographed football video, in which the football field image is in perspective. The right side is a normal rectangular football field. The process of converting the image on the left side into the image on the right side is called coordinate conversion. The pre-trained coordinate conversion model can convert the football field image on the left side into the normal rectangular image on the right side.

[0058] In one embodiment, the coordinate conversion of the to-be-processed video can be performed by a pre-trained coordinate conversion model, such as a coordinate conversion neural network, to obtain the coordinate-converted to-be-processed video.

[0059] S206, receiving the labeling result of the first frame of to-be-processed image uploaded by the client, identifying and tracking the target object in the to-be-processed video according to the labeling result and generating a target video, and sending the target video to the client.

[0060] In one embodiment, the coordinate conversion of the to-be-processed video can be performed by a pre-trained coordinate conversion model, such as a coordinate conversion neural network, to obtain the coordinate-converted to-be-processed video.

[0061] In one embodiment, the coordinate conversion of the to-be-processed video can be performed by a pre-trained coordinate conversion model, such as a coordinate conversion neural network, to obtain the coordinate-converted to-be-processed video.

[0062] In the aforementioned video processing method, the server first receives the video to be processed uploaded by the client, then performs coordinate transformation on the video, and sends the first frame of the transformed image to the client. The client then judges the coordinate transformation effect of the first frame and generates annotation results. This ensures the quality of the coordinate transformation of the video, thereby improving the accuracy of the subsequent target video generation. Next, the server tracks and identifies the target objects in the first frame based on the annotations uploaded by the client, generating a target video capable of simultaneously identifying and tracking the target objects.

[0063] In one embodiment, the above-mentioned identification and tracking of target objects in the video to be processed and generation of target video based on the annotation results includes: when the annotation results are qualified, identifying and tracking target objects in the coordinate-transformed video to be processed and generating target video; when the annotation results are unqualified, performing coordinate transformation on the video to be processed based on the coordinate information carried in the annotation results, and identifying and tracking target objects in the coordinate-transformed video to be processed and generating target video.

[0064] The annotation result refers to the result generated by the client judging the coordinate transformation effect of the first frame of the image to be processed received from the server, which may include two criteria: qualified and unqualified.

[0065] Specifically, when the annotation result from the server is satisfactory, the target object in the video to be processed after coordinate transformation is directly identified and tracked, and the target video is generated based on the identification and tracking results. When the annotation result received by the server is unsatisfactory, the coordinates of the video to be processed are transformed based on the coordinate information carried in the annotation result, and then the transformed video to be processed is identified and tracked, and the target video is generated based on the tracking results.

[0066] In one embodiment, the server can perform coordinate transformation based on the coordinate information and send the first frame of the video to be processed to the client. The client judges the coordinate transformation effect of the first frame of the video to be processed and generates the corresponding annotation result, which is then uploaded to the server. The server then performs coordinate transformation on the video to be processed based on the annotation result and sends the first frame of the video to the client again, until the annotation result received from the client is qualified.

[0067] In one embodiment, the client first uploads a football video to be processed to the server. The server performs coordinate transformation on the video, then analyzes it to obtain the first frame of the football image and sends it to the client. The user downloads the transformed football image, i.e., the first frame. The transformation effect of the first frame is then evaluated, and corresponding annotation results are generated. Continuing with... Figure 3As shown, if the preset number of points in the football video can correspond to the preset number of points in the right normal rectangle, it is judged to be qualified, otherwise the coordinate information is generated by the calibration program written by python (a programming software). The python calibration program combines the transformation of the preset number of points to calculate the pkl file (coordinate information), and then the client uploads the labeling result and coordinate information to the server for further analysis. Preferably, the preset number of points is four fixed points at the corners of the football field.

[0068] In the above embodiment, by taking the labeling result uploaded by the client as the effect of coordinate conversion of the to-be-processed video, and deciding whether to perform coordinate conversion again on the to-be-processed video according to the labeling result, the quality of coordinate conversion of the to-be-processed video can be ensured, and the accuracy of the generated target video can be ensured.

[0069] In one embodiment, the above-mentioned identification and tracking of the target object in the coordinate-converted to-be-processed video and generation of the target video include: identifying the target object in the to-be-processed video to obtain an action recognition image of each frame in the to-be-processed video; tracking the target object in the to-be-processed video to obtain a target tracking image of each frame in the to-be-processed video; matching each action recognition image and each target tracking image to obtain a target image of each frame; and connecting each target image to obtain the target video.

[0070] Among them, the action recognition object refers to the image generated after identifying the target object in the to-be-processed video; the target tracking image refers to the image generated after tracking the target object in the to-be-processed video; and the target image refers to the image generated after matching the target object in the action recognition image and the target tracking image.

[0071] Specifically, the server identifies the action of the target object in each frame of the to-be-processed image in the to-be-processed video to obtain an action recognition image of the target object in each frame of the to-be-processed image in the to-be-processed video, that is, the identification result mentioned above. Optionally, the target object can be identified by a pre-trained action recognition model, for example, PaStaNet (PaStaNet, an action recognition model).

[0072] Specifically, the server tracks the target object in each frame of the to-be-processed image in the to-be-processed video to obtain a target tracking image of the target object in each frame of the to-be-processed image in the to-be-processed video, that is, the tracking result mentioned above. Optionally, the target object in the to-be-processed image can be tracked by a pre-trained tracking model, for example, yolov5 (You only look once V5, a target detection model).

[0073] It should be noted that the order of identifying the target object in the to-be-processed video and tracking the target object in the to-be-processed video is not limited herein, that is, the target object in the to-be-processed video can be identified first or the target object in the to-be-processed video can be tracked first.

[0074] Specifically, after obtaining each frame of action recognition image and each frame of target tracking image in the to-be-processed video, the target objects in the corresponding frame of action recognition image and target tracking image are matched to obtain each frame of target image. Specifically, the target calculation value of the detection box in which the target object is located in the action recognition image and the target tracking image can be calculated, and the target object with the maximum target calculation value is matched. In an embodiment, the bounding box (detection box of the target object) obtained by the yolov5 network and the PaStaNet network is matched, the IOU (Intersection over Union, a standard for measuring the accuracy of detecting the corresponding object) value of the detection box of the target object between the corresponding frames is calculated respectively, and the detection box of the target object with the maximum IOU value is matched.

[0075] Finally, each frame of target image is connected to obtain a target video.

[0076] In an embodiment, the pasta network is used to obtain the action of each person at different time points, but cannot correspond each person (for example, the athlete B detects that the first frame A is doing action a, but detects that A' is doing action a in the second frame, and does not know whether the current A and A' are the same), the yolov5 performs target detection and tracking, and the tracked result is matched with the output of the pasta, because the yolov5 can know whether the A of the previous frame and the A' of the current frame are the same object, so that the identification and tracking can be realized at the same time.

[0077] In the above embodiment, the target object in the to-be-processed video is identified and tracked respectively to obtain each frame of action recognition image and each frame of target tracking image in the to-be-processed video, then each frame of action recognition image and each frame of target tracking image are matched to obtain each frame of target image, and finally the target images are connected to obtain a target video which can identify and track the target object at the same time.

[0078] In an embodiment, after the above matching each frame of action recognition image and each frame of target tracking image to obtain each frame of target image, the method further includes: calculating a feature value of each target object in the target image; and classifying each target object in the target image according to the feature value of the target object to obtain a category corresponding to the target object.

[0079] wherein the feature value is obtained according to information of a certain position in the detection box corresponding to the target object, and is a standard for classifying the target object.

[0080] Specifically, first, information of the center position of the detection box corresponding to each target object in each target image of each frame is calculated, and in this embodiment, color information of the center position of the detection box is calculated as the feature value, and in other embodiments, other information of other positions of the detection box can be calculated. After obtaining the feature value of each target object in each target image of each frame in the video to be processed, the feature values of the same target object in all frames are averaged as the feature value of the target object in the video, and in other embodiments, the feature value of the target object in the video can be obtained by calculating the variance or the mean square error. Then, the target object is classified according to the feature value to obtain the category corresponding to the target object. Exemplarily, taking a football video as an example, the target objects in the football video, i.e., the players, can be divided into teams according to the classification of the target objects according to the feature value. This facilitates subsequent analysis of the tactics in the football competition.

[0081] In one embodiment, before calculating the feature value of each target object in the target image, the target image is also subjected to noise reduction processing, wherein the noise reduction can be performed by Gaussian blur, so as to improve the signal-to-noise ratio of the target image and maximize the original information of the target image.

[0082] In one embodiment, the video to be processed is a football video, Gaussian blur is performed on each frame of picture (each frame of football target image), the color information (feature value) of the center position of the bounding box (detection box) of the player (target object) in each frame is sampled, and the color mean value of the player is obtained as the color feature vector of the player. The obtained color vector of the player is subjected to k-means binary clustering to obtain team division information. In this way, various analyses of the football video can be performed, such as player skill and tactics (according to action recognition), player physical fitness (according to tracking the running route of the player), and the purpose of performing these analyses is to facilitate subsequent designation of tactics by the coach.

[0083] In the above embodiment, the feature value of each target object in the target image is first calculated, which is to take the feature value to represent the detection box in which the target object is located, so as to facilitate classification; the target object is classified according to the feature value of the target object, so as to facilitate further processing of the target video, such as football tactics analysis.

[0084] In one embodiment, as shown in Figure 4 , a video processing method is provided, which is taken as an example of the server 102 in Figure 1 for illustration, and includes the following steps:

[0085] S402, send the to-be-processed video to the server.

[0086] S404, receive the first frame of to-be-processed image after coordinate conversion sent by the server, and judge the first frame of to-be-processed image after coordinate conversion, generate a labeling result, and send the labeling result to the server; the first frame of to-be-processed image is obtained by the server through coordinate conversion on the to-be-processed video.

[0087] Specifically, the client receives the first frame of to-be-processed image after coordinate conversion sent by the server, and judges the first frame of to-be-processed image, that is, analyzes the quality of coordinate conversion of the first frame of to-be-processed image. If the preset number of points in the football field in the first frame of football video can correspond to the preset number of points in the right normal rectangle, it is judged to be qualified, otherwise the coordinate information is generated by the calibration program written by python. The python calibration program calculates the pkl file (coordinate information) combined with the transformation of the preset number of points, and then the client uploads the labeling result and the coordinate information to the server for further analysis.

[0088] S406, receive the target video sent by the server, and the target video is generated by the server according to the labeling result to identify and track the target object in the to-be-processed video.

[0089] In one embodiment, in combination with Figure 5 as shown, Figure 5 is a schematic view of a certain frame in the target video in one embodiment. Specifically, the left field is the transformed field coordinate, and a player of a team is displayed. The numbers in the figure represent the numbers of the players, and the positions of the numbers represent the actual positions of the players after transformation in the video. The right side is the pose estimation of the players by the coordinate transformation neural network, wherein the ID field represents the number of the player, corresponding to the number on the left side of the figure, and the action below the ID represents the part-level pose estimation of the player by PaStaNet, that is, the pose estimation of each body part.

[0090] In one embodiment, the user downloads the mp4 file (target video) and opens the analysis part of the GUI for viewing. The analysis GUI supports frame-by-frame pause and reading and writing the player action and team tactics of the frame. If you want to manually correct, you can also click to modify and save for next viewing and research.

[0091] In the above embodiment, the client first uploads the to-be-processed video, then receives the first frame of to-be-processed image after coordinate conversion sent by the server, then performs coordinate conversion on the first frame of to-be-processed image, generates a labeling result, and uploads the server, so as to instruct the server to continue coordinate conversion on the to-be-processed image or to identify and analyze the target object in the to-be-processed image, and finally receives the target video sent by the server, the target video being a video after identification and tracking of the target object in the to-be-processed image. After the client receives the target video, the user can further process the target video.

[0092] In one embodiment, a GUI (graphical user interface) is also designed, which combines Figure 6 as shown, Figure 6 is a schematic diagram of the running process of the GUI in one embodiment. First, the client first uploads the video to be analyzed (to-be-processed video), and after receiving the connection request, the server downloads the video and starts the neural network for analysis. After the analysis is completed, the server notifies the client to download the labeled data, and after the data download is completed, the client can interact with the server to complete the re-analysis and other tasks, further improving the accuracy. Wherein User GUI is the client GUI, Server is the server, video is the video, and labeled data is the data label. Figure 7 and Figure 8 is a schematic diagram of the GUI display in other embodiments.

[0093] In the above embodiment, the to-be-processed video can be processed more directly and intuitively through the GUI interface.

[0094] In one embodiment, Qt6 (a program development framework) is used to develop the client GUI and server framework. The data to be displayed in this embodiment includes: football video, labeled video, player position video, and server information. For video playback, it needs to be accurate to the frame. Qt6 provides a multimedia module QMediaPlayer (Qt's own multimedia) that supports video dragging and can pause frame by frame. Therefore, in the video playback part, this embodiment mainly uses QMediaPlayer, supplemented by a video control module; in the text display part, this embodiment mainly uses QJson+QTableWidget, supplemented by file parsing and reading and writing.

[0095] In the above embodiment, Qt6 and QMediaPlayer can be used to process the target video subsequently, such as tactical analysis of the target video.

[0096] In one embodiment, the end interaction must implement the client sending commands, the server receiving commands and executing, and then returning the results to the client. Therefore, the end interaction must meet the real-time and reliability requirements, so in this embodiment, the QProcess module is used in conjunction with the Connection module written in this embodiment to achieve this requirement. When executing the command, the server first creates a QProcess instance, initializes the command to it, and then runs the process. When QProcess has output that can be read in, a signal is released. The Connection module accepts the signal, then reads the output from the QProcess instance to the Connection, and then sends the Connection to the client's Connection, which is finally displayed in the text editing field.

[0097] In one embodiment, the file transmission between the client and the server inherits the design idea of FTP, that is, multiple connections are opened, one for transmitting instructions and multiple for transmitting data. Both the file and instruction transmission modules are written based on the transmission layer API QTcpSocket (a kind of socket) provided by Qt, and can achieve reliable transmission. For file transmission, the group implements the Qt4.0 protocol for file transmission, including the greeting between the server and the client at the beginning, the inquiry and confirmation for each metadata transmission, and the check when the file transmission is completed. For instruction transmission, the group also derives from the underlying QTcpServer and finally writes the Connection class to implement all functions. The specific transmission protocol and transmission results are shown in Figure 9 , Figure 9 is a QT transmission diagram in one embodiment. The data transmission mode of the server and the client is reliable transmission, and the process is: the client sends a connection request to the server, the server agrees to the client's connection request after receiving the request and then returns a greeting, the client receives the greeting and starts to transmit data to the server, the server sends an ACK signal to the client after receiving the data, transfering represents the transmission process of the first data to the last data, the client sends an OK signal to the server after the last data transmission is completed to tell the server that the data has been sent, the server sends a bye signal to the client after receiving the signal, the client agrees to disconnect and then returns a bye signal to the server, and the server continues to listen to the connection request from other clients.

[0098] In the above embodiment, the client and server are separated and connected by TCP (Transmission Control Protocol), which facilitates the subsequent deployment of this project in the cloud. This makes good use of computing resources and simplifies management. Furthermore, since the computing end needs to run the neural network in a Linux environment, the separation of the user end and the computing end eliminates the inconvenience of users configuring the environment.

[0099] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0100] Based on the same inventive concept, this application also provides a video processing apparatus for implementing the video processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more video processing apparatus embodiments provided below can be found in the limitations of the video processing method described above, and will not be repeated here.

[0101] In one embodiment, such as Figure 10 As shown, a video processing device for a server is provided, comprising: a receiving module 100, a coordinate transformation module 200, and a recognition module 300, wherein:

[0102] The receiving module 100 is used to receive the video to be processed uploaded by the client.

[0103] The coordinate transformation module 200 is used to perform coordinate transformation on the video to be processed and send the first frame of the image to be processed after coordinate transformation to the client.

[0104] The recognition module 300 is used to receive the annotation results of the first frame of the image to be processed uploaded by the client, identify and track the target objects in the video to be processed according to the annotation results, generate the target video, and send the target video to the client.

[0105] In one embodiment, the identification module 300 includes:

[0106] The labeling qualification submodule is used to identify and track target objects in the coordinate-transformed video to be processed and generate target video when the labeling result is qualified.

[0107] The "Unqualified Labeling" submodule is used to perform coordinate transformation on the video to be processed based on the coordinate information carried in the labeling result when the labeling result is unqualified, and to identify and track the target object in the coordinate transformed video to be processed and generate the target video.

[0108] In one embodiment, the above-mentioned qualified labeling submodule and the above-mentioned unqualified labeling submodule include:

[0109] An image recognition unit is used to identify target objects in the video to be processed, and to obtain an action recognition image for each frame of the video to be processed.

[0110] The image tracking unit is used to track target objects in the video to be processed, and obtain target tracking images for each frame of the video to be processed.

[0111] The image matching unit is used to match each frame of the action recognition image with each frame of the target tracking image to obtain each frame of the target image.

[0112] The stitching unit is used to connect each frame of the target image to obtain the target video.

[0113] In one embodiment, the above-mentioned qualified labeling submodule and the above-mentioned unqualified labeling submodule include:

[0114] The feature value calculation unit is used to calculate the feature value of each target object in the target image.

[0115] The classification unit is used to classify each target object in the target image according to the feature value of the target object, and obtain the category corresponding to the target object.

[0116] In one embodiment, such as Figure 11 As shown, a video processing apparatus for a server is provided, comprising: a sending module 400, a label receiving module 500, and a target video receiving module 600, wherein:

[0117] The sending module 400 is used to send the video to be processed to the server;

[0118] The labeling receiving module 500 is configured to receive the first frame of to-be-processed image after coordinate conversion sent by the server, judge the first frame of to-be-processed image after coordinate conversion, generate a labeling result, and send the labeling result to the server; the first frame of to-be-processed image is obtained by performing coordinate conversion on the to-be-processed video by the server;

[0119] The target video receiving module 600 is configured to receive a target video sent by the server, wherein the target video is generated by identifying and tracking a target object in the to-be-processed video according to the labeling result by the server.

[0120] The above-mentioned modules in the video processing apparatus can be realized by software, hardware and combinations thereof in whole or in part. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to the above-mentioned modules.

[0121] In one embodiment, a computer device is provided, which can be a server, and an internal structure diagram thereof can be as shown in Figure 12 The computer device includes a processor, a memory and a network interface connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the computer device is configured to store to-be-processed videos. The network interface of the computer device is configured to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement a video processing method.

[0122] Those skilled in the art can understand that Figure 12 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0123] In one embodiment, a computer device is provided, which includes a memory and a processor, the memory stores a computer program, and the processor executes the computer program to implement the steps of the method in any one of the above-mentioned embodiments.

[0124] In one embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the steps of the method in any one of the above-mentioned embodiments.

[0125] In one embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the steps of the method of any one of the above embodiments.

[0126] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiments of the method. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0127] The technical features of the above embodiments can be combined in any manner. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist, they should be considered as the scope of the present disclosure.

[0128] The above-described embodiments are merely illustrative of several embodiments of the present application, and the description is relatively specific and detailed, but should not be understood as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. A video processing method, characterized in that, Applied to the server side, the method includes: Receive videos uploaded by the client that are yet to be processed; Perform coordinate transformation on the video to be processed, and send the first frame of the image to be processed after coordinate transformation to the client. The system receives the annotation results of the first frame image to be processed uploaded by the client, identifies and tracks the target objects in the video to be processed based on the annotation results, generates a target video, and sends the target video to the client. The step of identifying and tracking target objects in the video to be processed based on the annotation results and generating a target video includes: When the annotation result is qualified, the target object in the video to be processed after coordinate transformation is identified and tracked and the target video is generated. When the annotation result is unqualified, the coordinate transformation of the video to be processed is performed according to the coordinate information carried in the annotation result, and the target object in the coordinate transformed video to be processed is identified and tracked to generate a target video. The process of identifying and tracking target objects in the coordinate-transformed video and generating a target video includes: The target object in the video to be processed is identified to obtain the action recognition image of each frame of the video to be processed; The target object in the video to be processed is tracked to obtain the target tracking image of each frame of the video to be processed; The action recognition image and the target tracking image in each frame are matched to obtain the target image in each frame; Each frame of the target image is concatenated to obtain the target video; The step of matching each frame of the action recognition image and each frame of the target tracking image to obtain each frame of the target image includes: Calculate the IOU value of the detection box corresponding to the target object in each frame of the action recognition image and the target object in each frame of the target tracking image; The motion recognition image and the target tracking image in each frame are matched according to the IOU value to obtain the target image in each frame.

2. The method according to claim 1, characterized in that, After matching each frame of the action recognition image and each frame of the target tracking image to obtain each frame of the target image, the method further includes: Calculate the feature value of each target object in the target image; For each target object in the target image, the target object is classified according to its feature values ​​to obtain the category corresponding to the target object.

3. A video processing method, characterized in that, Applied to a client, the method includes: Send the video to be processed to the server; The system receives a first frame of the image to be processed after coordinate transformation sent by the server, judges the first frame of the image to be processed after coordinate transformation, generates a labeling result, and sends the labeling result to the server; the first frame of the image to be processed is obtained by the server performing coordinate transformation on the video to be processed; after receiving the labeling result, the server identifies and tracks the target object in the video to be processed according to the labeling result and generates a target video; the server identifies and tracks the target object in the video to be processed according to the labeling result and generates a target video, including: when the labeling result is qualified, identifying and tracking the target object in the video to be processed after coordinate transformation and generating a target video; when the labeling result is unqualified, performing coordinate transformation on the video to be processed according to the coordinate information carried in the labeling result, identifying and tracking the target object in the video to be processed after coordinate transformation and generating a target video; The server identifies and tracks target objects in the coordinate-transformed video to generate a target video, including: identifying target objects in the video to obtain motion recognition images for each frame; tracking target objects in the video to obtain target tracking images for each frame; matching each frame of motion recognition images with each frame of target tracking images to obtain a target image; and concatenating each frame of target images to obtain the target video. The server matching each frame of motion recognition images with each frame of target tracking images to obtain a target image includes: calculating the Interchange of Union (IOU) value of the bounding boxes corresponding to the target objects in each frame of motion recognition images and each frame of target tracking images; and matching each frame of motion recognition images with each frame of target tracking images based on the IOU value to obtain the target image. The server receives a target video sent by the server, which is generated by the server to identify and track target objects in the video to be processed based on the annotation results.

4. A video processing apparatus, characterized in that, Applied to the server side, the device includes: The receiving module is used to receive videos uploaded by the client that are yet to be processed. The coordinate transformation module is used to perform coordinate transformation on the video to be processed and send the first frame of the image to be processed after coordinate transformation to the client. The recognition module is used to receive the annotation results of the first frame image to be processed uploaded by the client, identify and track the target objects in the video to be processed according to the annotation results and generate a target video, and send the target video to the client; The identification module includes: The labeling qualification submodule is used to identify and track the target object in the coordinate-transformed video to be processed and generate the target video when the labeling result is qualified. The "Unqualified" submodule is used to perform coordinate transformation on the video to be processed based on the coordinate information carried in the annotation result when the annotation result is unqualified, and to identify and track the target object in the coordinate transformed video to be processed and generate the target video. The identification module further includes: An image recognition unit is used to identify target objects in the video to be processed, and to obtain an action recognition image for each frame of the video to be processed. An image tracking unit is used to track target objects in the video to be processed, and obtain target tracking images for each frame of the video to be processed. An image matching unit is used to match each frame of the action recognition image with each frame of the target tracking image to obtain each frame of the target image; and to connect each frame of the target image to obtain the target video; The recognition module is further configured to calculate the IOU value of the detection box corresponding to the target object in each frame of the action recognition image and the target object in each frame of the target tracking image; and to match each frame of the action recognition image and each frame of the target tracking image according to the IOU value to obtain each frame of the target image.

5. The apparatus according to claim 4, characterized in that, The identification module further includes: A feature value calculation unit is used to calculate the feature value of each target object in the target image; The classification unit is used to classify each target object in the target image according to the feature value of the target object, and obtain the category corresponding to the target object.

6. A video processing apparatus, characterized in that, For a client, the device includes: The sending module is used to send the video to be processed to the server; The annotation receiving module is used to receive the first frame of the image to be processed after coordinate transformation sent by the server, judge the first frame of the image to be processed after coordinate transformation, generate annotation results, and send the annotation results to the server; the first frame of the image to be processed is obtained by the server performing coordinate transformation on the video to be processed; after receiving the annotation results, the server identifies and tracks the target objects in the video to be processed according to the annotation results and generates a target video; the server identifies and tracks the target objects in the video to be processed according to the annotation results and generates a target video, including: when the annotation results are qualified, identifying and tracking the target objects in the video to be processed after coordinate transformation and generating a target video; when the annotation results are unqualified, performing coordinate transformation on the video to be processed according to the coordinate information carried in the annotation results, identifying and tracking the target objects in the video to be processed after coordinate transformation and generating a target video. The server identifies and tracks target objects in the coordinate-transformed video to generate a target video, including: identifying target objects in the video to obtain motion recognition images for each frame of the video; tracking target objects in the video to obtain target tracking images for each frame of the video; matching each frame of motion recognition images with each frame of target tracking images to obtain a target image for each frame; and concatenating each frame of target images to obtain the target video. The server matching each frame of motion recognition images with each frame of target tracking images to obtain a target image for each frame includes: calculating the Interchange of Union (IOU) value of the bounding boxes corresponding to the target objects in each frame of motion recognition images and each frame of target tracking images; and matching each frame of motion recognition images with each frame of target tracking images based on the IOU value to obtain the target image for each frame. The target video receiving module is used to receive the target video sent by the server. The target video is generated by the server to identify and track target objects in the video to be processed based on the annotation results.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 2 or 3.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 2 or 3.

9. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 2 or 3.

Images