A video data processing method and related device

Abstract

The present application provides a video data processing method and related devices. The method includes: obtaining video data of a competitive game; tracking the target of the video data to determine the position information of the target object at each moment; creating a virtual object corresponding to the target object in a virtual scene based on the position information, and the virtual scene is created based on the competition venue of the competitive game; adding a virtual perspective to the virtual object; and displaying the picture of the virtual scene under the virtual perspective in response to the perspective selection operation of the target object. The method provided in the embodiment of the present application can be used to improve the experience of video viewers.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a video data processing and related devices. Background Art

[0002] With the development of artificial intelligence technology, 3D modeling can be performed through video data to achieve a high degree of restoration of the scene. The emergence of this technology provides us with a new way to present and record various scenes in the real world. For highly interactive video content, how to improve the audience's immersion and sense of participation is still an important issue facing technology developers. Summary of the invention

[0003] The embodiments of the present application provide a video data processing method and related devices to improve the experience of video viewers.

[0004] One aspect of the present application provides a video processing method, comprising:

[0005] Obtain video data of competitive games;

[0006] Performing target tracking on the video data to determine the position and posture information of the target object at each moment;

[0007] Based on the position information, creating a virtual object corresponding to the target object in a virtual scene, where the virtual scene is created based on a competition venue;

[0008] adding a virtual perspective to the virtual object;

[0009] In response to the perspective selection operation on the target object, a picture of the virtual scene under the virtual perspective is displayed.

[0010] In a possible implementation method, obtaining video data includes:

[0011] Acquiring the video data collected by a fixed shooting device;

[0012] The target tracking of the video data to determine the position information of the target object at each moment includes:

[0013] Analyze the video data frame by frame to determine the target image coordinates of the target object at each moment in the image coordinate system;

[0014] Based on the calibration parameters of the photographing device, the target image coordinates are converted into target world coordinates in a world coordinate system;

[0015] The position information is determined based on the target world coordinates at each moment.

[0016] In a possible implementation method, before performing target tracking on the video data, the method further includes:

[0017] Perform moving target detection on the video data, and determine the moving object in the video data as the target object.

[0018] In a possible implementation method, before converting the target image coordinates into target world coordinates in a world coordinate system based on the calibration parameters of the shooting device, the method further includes:

[0019] Obtaining fixed world coordinates of a fixed object in the video data in the world coordinate system;

[0020] Determining the fixed image coordinates of the fixed object in the image coordinate system through the video data;

[0021] The calibration parameters are calculated using the fixed world coordinates and the fixed image coordinates.

[0022] In a possible implementation method, analyzing the video data frame by frame to determine the target image coordinates of the target object at each moment in the image coordinate system includes:

[0023] Analyze the video data frame by frame to determine the target image coordinates of the target object in each frame image;

[0024] The target image coordinates of the target object at each moment are calculated based on the frame rate of the video data.

[0025] In a possible implementation method, obtaining video data includes:

[0026] The video data collected by multiple shooting devices at different positions is obtained; the posture information is specifically the posture information in a three-dimensional coordinate system.

[0027] In a possible implementation method, creating a virtual object corresponding to the target object in a virtual scene based on the posture information includes:

[0028] Creating the virtual scene based on the fixed objects in the video data;

[0029] creating the virtual object based on the target object;

[0030] Based on the posture information, determining the position form of the virtual object at each moment in the virtual scene;

[0031] Based on the position form, the virtual object is set in the virtual scene.

[0032] In a possible implementation method, after creating a virtual object corresponding to the target object in a virtual scene based on the posture information, the method further includes:

[0033] In response to the role adding request, creating a virtual role in the virtual scene;

[0034] A character perspective is added to the virtual character, and a picture of the virtual scene from the character perspective is returned.

[0035] In a possible implementation method, before creating a virtual character in the virtual scene in response to the character adding request, the method further includes:

[0036] An interactive element is created in the virtual scene, and the interactive element is used to interact with the virtual character in response to an interactive operation of the virtual character.

[0037] In a possible implementation method, the virtual scene and the virtual object are created by an Unreal Engine.

[0038] In a possible implementation method, after displaying a picture of a virtual scene under a virtual perspective in response to a perspective selection operation on a target object, the method further includes:

[0039] In response to the viewing angle adjustment operation, the viewing angle of the picture is changed; the viewing angle adjustment operation includes a viewing angle translation operation, a viewing angle rotation operation, and a viewing angle zoom operation.

[0040] Another aspect of the present application provides a video data processing device, comprising:

[0041] Acquisition module, used to acquire video data of competitive games;

[0042] The posture calculation module is used to track the target in the video data and determine the posture information of the target object at each moment;

[0043] A modeling module, used to create a virtual object corresponding to the target object in a virtual scene based on the posture information, and the virtual scene is created based on the competition venue of the competition;

[0044] A perspective adding module, used to add a virtual perspective to a virtual object;

[0045] The display module is used to display the image of the virtual scene under the virtual perspective in response to the perspective selection operation of the target object.

[0046] In a possible implementation method, the acquisition module is specifically used to acquire video data collected by a fixed shooting device;

[0047] The posture calculation module is specifically used to determine the target image coordinates of the target object at each moment in the image coordinate system; based on the calibration parameters of the shooting device, the target image coordinates are converted into the target world coordinates in the world coordinate system; based on the target world coordinates at each moment, the posture information is determined.

[0048] In a possible implementation method, the method further includes:

[0049] The target detection module is used to perform moving target detection on the video data and determine the moving object in the video data as the target object.

[0050] In a possible implementation method, the method further includes:

[0051] The calibration calculation module is used to obtain the fixed world coordinates of the fixed object in the video data in the world coordinate system; determine the fixed image coordinates of the fixed object in the image coordinate system through the video data; and calculate the calibration parameters through the fixed world coordinates and the fixed image coordinates.

[0052] In a possible implementation method, the posture calculation module is specifically used to analyze the video data frame by frame to determine the target image coordinates of the target object in each frame image; based on the frame rate of the video data, the target image coordinates of the target object at each moment are calculated.

[0053] In a possible implementation method, the acquisition module is specifically used to acquire video data collected by multiple shooting devices at different positions; the posture information is specifically posture information in a three-dimensional coordinate system.

[0054] In a possible implementation method, the modeling module is specifically used to create a virtual scene based on fixed objects in video data; create a virtual object based on a target object; determine the position and form of the virtual object at each moment in the virtual scene based on posture information; and set the virtual object in the virtual scene based on the position and form.

[0055] In a possible implementation method, the method further includes:

[0056] The role adding module is used to create a virtual role in a virtual scene in response to a role adding request; add a role perspective to the virtual role, and return a picture of the virtual scene under the role perspective.

[0057] In a possible implementation method, the method further includes:

[0058] The element adding module is used to create interactive elements in the virtual scene, and the interactive elements are used to respond to the interactive operations of the virtual characters and interact with the virtual characters.

[0059] In a possible implementation method, the virtual scene and the virtual objects are created by Unreal Engine.

[0060] In a possible implementation method, the method further includes:

[0061] The viewing angle adjustment module is used to change the viewing angle of the picture in response to the viewing angle adjustment operation; the viewing angle adjustment operation includes the viewing angle translation operation, the viewing angle rotation operation and the viewing angle zoom operation.

[0062] Another aspect of the present application provides a computer device, comprising: a memory, and a processor;

[0063] The memory stores instructions, and when the instructions are executed on the processor, the above methods are executed.

[0064] Another aspect of the present application provides a computer-readable storage medium, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer is enabled to execute the above-mentioned methods.

[0065] Another aspect of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided by the above aspects.

[0066] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0067] The present application provides a video data processing method and related devices. First, a virtual scene and a virtual object are created based on the video data of a competitive game. The virtual object is a target object in the video. Then, a virtual perspective is added to the virtual object in the virtual scene. The virtual scene is created based on the competition venue of the competitive game. When a user wants to have an immersive experience from the perspective of the target object, the field of view of the target object can be selected through a corresponding terminal device. After responding to the perspective selection operation, the image of the virtual scene under the virtual perspective is rendered, so that the user can bring himself into the role to experience it, feel as if he is in the scene, and obtain an immersive experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 This is an application environment diagram of the video data method in the embodiment of the present application;

[0069] Figure 2 A method flow chart of a video data processing method provided in an embodiment of the present application;

[0070] Figure 3 A method flow chart of a video data processing method provided in an embodiment of the present application;

[0071] Figure 4 A technical flow chart of the video data processing method provided in the embodiment of the present application;

[0072] Figure 5a to Figure 5c Schematic diagram of each coordinate system;

[0073] Figures 6a to 6d It is a schematic diagram of the transformation between various coordinate systems;

[0074] Figure 7 A product flow chart of the video data processing method provided in the embodiment of the present application;

[0075] Figures 8a to 8d A diagram showing a practical application of the video data processing method provided in an embodiment of the present application;

[0076] Fig. 9 This is a schematic diagram of an embodiment of a video data processing device 900 in an embodiment of the present application;

[0077] Fig.10 It is a schematic diagram of a server structure provided in an embodiment of the present application. DETAILED DESCRIPTION

[0078] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "corresponding to" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0079] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0080] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0081] Computer Vision (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision that uses cameras and computers to replace human eyes to identify and measure targets, and further performs image processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, etc.

[0082] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common game development, competitive events (including game competitions, sports competitions, etc.), smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0083] The solution provided in the embodiments of the present application involves computer vision technology, mainly including image processing, video processing, 3D technology, etc. The specific examples are described below.

[0084] With the development of artificial intelligence technology, the application areas of video data are also continuously expanding. Through the combination of deep learning and computer vision technologies, it is now possible to use video data for accurate 3D modeling. This modeling method can achieve a high degree of restoration of the scene, making the reconstructed 3D scene lifelike.

[0085] However, for highly interactive video content, simple 3D modeling is not enough to satisfy the audience's pursuit of a sense of opportunity and participation. How to improve the audience's sense of immersion and participation, so that they can more deeply integrate into the video content and interact with the characters or environment in it, has become an important issue faced by technology developers.

[0086] In order to solve the above problems, the embodiment of the present application provides a video data processing method and related devices, which are applied to the video data of competitive games. First, the video data is tracked to determine the position information of the target object in the video content at each moment; then, based on the position information, a virtual image (corresponding to the virtual object) with the target object is created in the virtual scene. Since the competition venue of the competitive game is a standard venue, the virtual scene can be created based on the competition venue; then in the virtual scene, a perspective is added to the virtual object (corresponding to the virtual perspective). When the audience watches the video, they can click on the perspective of the target object to show the audience the picture of the virtual scene under the virtual perspective. This method enables the audience to enter the virtual scene constructed based on the video data, and then watch the content in the video from a first-person perspective, so that the audience can bring in the role to experience it, and have an immersive feeling, thereby improving the audience's experience and sense of participation.

[0087] For easier understanding, see Figure 1 , Figure 1 FIG. 1 is an application environment diagram of the video data method in the embodiment of the present application, such as Figure 1 As shown, the video data method in the embodiment of the present application is applied to a video data system. The video data system includes: a server and a terminal device; wherein the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The terminal device and the server can be directly or indirectly connected via wired or wireless communication, and the embodiments of the present application are not limited here.

[0088] The video processing method provided in the embodiment of the present application can be executed by a server. The server first obtains the video data of the competitive game, which can be collected by a terminal device and stored in the server. The server then tracks the target of the video data and determines the position information of the target object at each moment. The position information includes coordinates, speed, acceleration, etc. After determining the position information, based on the position information, a virtual object corresponding to the target object is created in a virtual scene created based on the competition venue of the competitive game, and then a virtual perspective is added to the virtual object; after the audience selects the perspective of the target object, the picture in the virtual scene viewed through the virtual perspective is displayed.

[0089] The following is an introduction to the video data processing method in this application from the perspective of the server as the execution subject. Figure 2 , Figure 2 The method flow chart of the video data processing method provided in the embodiment of the present application includes:

[0090] 201, obtaining video data of a competitive game.

[0091] It is understandable that in order to process video data, the video data needs to be acquired first.

[0092] There are many ways to obtain video data. Here are some common methods:

[0093] If the video data is pre-stored on a local computer or server, you can directly use the file path to read the video data; if the video data is located on the Internet, you can use the network request library to download the video file. After the download is complete, you can process the video like a local file; if the video data is transmitted via real-time streaming, such as camera capture, webcam, etc., you can use appropriate libraries or application programming interfaces (Application Programming Interface, API) to capture these streams; in some cases, video data may be stored in a database or a specific storage system, and you need to write corresponding code to extract video data from these systems.

[0094] It can be understood that the video data processing method provided in the embodiment of the present application can be applied to the scenario of event playback, that is, the video data is an event video of a competitive game, and the event video is usually shot by multiple shooting devices, so the video data can specifically be video data collected by multiple shooting devices.

[0095] 202, tracking the target on the video data to determine the position and posture information of the target object at each moment.

[0096] After acquiring the video data, the target is tracked to determine the position and posture information of the target object at each moment. The position and posture information includes the position (such as coordinates) and posture (such as direction, angle, acceleration, etc.) of the target object in three-dimensional space.

[0097] The following are some common methods and techniques for determining pose information:

[0098] 1) Use feature point detection algorithms to detect feature points of target objects in video frames; match these feature points between consecutive frames to track the movement of target objects; and estimate the position and posture of target objects by analyzing the displacement and changes of feature points.

[0099] 2) By calculating the displacement of pixels between adjacent frames, the motion trajectory of the target object in the entire video sequence can be tracked; based on the motion trajectory of the target object, the position and posture changes of the target object can be further analyzed.

[0100] 3) Use target detection algorithms (such as YOLO-v8x) to detect target objects in each frame; use target tracking algorithms (such as Bytetracking) to associate and track detected target objects between consecutive frames; and extract the pose information of the target object by analyzing the position and size of the target bounding box.

[0101] 4) Use deep neural networks (such as convolutional neural networks (CNNs)) to extract features of the target object and learn how to track these features by training the model; some end-to-end deep learning models can directly predict the position and pose of the target object from video data.

[0102] For the calculation results of the pose information, an optimization algorithm can be used to smooth and correct the pose trajectory. The optimization algorithm can help deal with problems such as noise, occlusion, and dynamic changes, and improve the accuracy of the pose information calculation.

[0103] 203 , based on the position information, create a virtual object corresponding to the target object in a virtual scene, where the virtual scene is created based on a competition venue.

[0104] A virtual scene refers to a non-real environment or space constructed by technical means, and a virtual object refers to a non-real entity created by digital technology, which has a realistic image and behavior and can be displayed and interacted in a virtual scene. Through virtual scenes and virtual objects, users can be provided with an immersive experience.

[0105] The virtual scene can be created based on the competition venue of the competition. It is understandable that the competition venue usually has certain specifications and standards, so the virtual scene can be pre-established based on the standard scene. For example, when the video data is a video of a football match, the virtual scene can include grass, a goal, and marking lines used in football matches on the grass. Further, spectator seats can be set up; when the video data is a video of a game competition, the virtual scene is a pre-set game scene, which can include roads, grass, rivers, walls, obstacles, etc.

[0106] Virtual scenes can be created through 3D modeling software or game engines. Various objects, textures, and lighting conditions can be arranged in virtual scenes to simulate real environments. A virtual object corresponding to the target object is created in the virtual scene. The virtual object should have similar geometric shapes, textures, or appearance features to the target object.

[0107] After obtaining the pose information, the pose information is applied to the virtual object, including updating the position and rotation angle of the virtual object in the virtual scene in each frame to match the position and pose of the target object in the video.

[0108] 204, adding a virtual perspective to the virtual object.

[0109] When the virtual scene and virtual objects are arranged, a virtual perspective can be added to the virtual object. The virtual perspective is equivalent to a camera or observation point, which will represent the perspective of the target object. The camera is associated with the virtual object, so that when the virtual object moves or rotates, the camera's perspective will be updated accordingly, that is, the camera becomes a child object or appendage of the virtual object, so that it follows its movement.

[0110] 205 , in response to the perspective selection operation on the target object, displaying the image of the virtual scene under the virtual perspective.

[0111] When the virtual perspective is created, the user can select the perspective of the target object when watching the video data, so as to use the rendering engine in the virtual scene to render the scene from the camera perspective. The rendering result is output to a display, projector or other display device. Since the virtual object is created based on the position information of the target object, the user can observe the visual experience from the perspective of the target object through virtual.

[0112] There are many ways for users to select the perspective of the target object, such as clicking on a virtual object, using shortcut keys, or through gesture recognition of VR devices. When there are multiple virtual objects, after detecting the perspective selection operation, identify the perspective of the target object that the user wants to switch to, and then activate the virtual perspective of the corresponding virtual object, which means binding the previously set camera or observation point to the selected target object. If the virtual object itself can move or rotate, the camera should also follow its movement to maintain the consistency of the perspective.

[0113] Once the camera is bound to the virtual object, the system needs to update the rendering process to render the virtual scene from the new perspective (i.e. the perspective of the virtual object). This involves adjusting the camera's position, orientation, and field of view to ensure that the image accurately reflects the perspective of the virtual object. The rendered image from the virtual perspective is displayed to the user through a monitor, VR helmet, or other display device.

[0114] The user may want to switch back to other perspectives or perform other interactive operations at any time. After receiving such operation requests, the perspective and corresponding rendering are updated accordingly. In addition, when the user wants to control the field of view, in response to the user's field of view movement and selection operations through input devices (such as mouse, keyboard or VR controller), the camera is moved and selected accordingly, so that the user can watch the video from the field of view of different target objects.

[0115] The video data processing method provided in the embodiment of the present application first creates a virtual scene and a virtual object based on the video data of a competitive game, where the virtual object is the target object in the video; then a virtual perspective is added to the virtual object in the virtual scene, where the virtual scene is created based on the competition venue of the competitive game. When a user wants to have an immersive experience from the perspective of the target object, the user can select the field of view of the target object through a corresponding terminal device, and after responding to the perspective selection operation, the image of the virtual scene under the virtual perspective is rendered, so that the user can experience the role, feel personally on the scene, and obtain an immersive experience.

[0116] In this application Figure 2 In an optional embodiment of the video data processing method provided in the corresponding embodiment, please refer to Figure 3 , Figure 3 A method flow chart of a video data processing method provided in an embodiment of the present application.

[0117] For ease of understanding, let's combine Figure 4 The method flow chart of the video data processing method provided in the embodiment of the present application is described. Figure 4 A technical flow chart of the video data processing method provided in an embodiment of the present application.

[0118] like Figure 3 As shown, the method of the embodiment of the present application includes:

[0119] 301, obtaining video data collected by a fixed shooting device;

[0120] It is understandable that the above step 301 corresponds to Figure 2 Step 201 in the illustrated embodiment.

[0121] In this embodiment, the video data is acquired by a fixed shooting device. When the video data is a video of some sports events (such as basketball games, football games, etc.), a camera fixed in the venue can be used to record the game. Figure 4 Capture video stream through camera in.

[0122] The number of the imaging device may be one, but is preferably multiple.

[0123] When the number of shooting devices is one, the corresponding posture information is also on a two-dimensional plane, which is suitable for billiards, bowling and other games played on a plane. In most cases, the movement of objects in the vertical direction can be ignored in such games, so there is no need to consider it, which can simplify the acquisition and processing of posture information. For such video data, it is still meaningful to create virtual scenes and virtual objects. The virtual scene can simulate the competition venue, while the virtual objects represent the players, referees and other people who actually participate in the game. These virtual objects can move synchronously according to the posture information extracted from the video data to simulate the actual game situation. When the audience chooses the player's perspective to watch the game, the system can bind the camera to the virtual object of the selected player, so as to display the game screen from the player's perspective. The audience can observe the angle, action and surrounding environment of the player's batting, providing a more immersive viewing experience. Similarly, when the user chooses the referee's perspective, the system can bind the camera to the referee's virtual object. The audience can observe the game process from the referee's perspective, including the player's actions, the trajectory of the ball and the referee's decision-making process for various judgments. This perspective switching function helps viewers better understand the rules of the game and the basis for the referee's decision-making, and can also achieve an immersive experience.

[0124] When there are multiple photographing devices, step 301 specifically includes:

[0125] Video data collected by multiple shooting devices at different positions are obtained; the posture information is specifically the posture information in a three-dimensional coordinate system.

[0126] It is understandable that when there are multiple cameras, video data collected by multiple cameras at different positions can be obtained, and these pose information will be specifically expressed as pose information in a three-dimensional coordinate system. This situation provides a rich data source for creating more accurate and comprehensive virtual scenes and virtual objects, especially for those games or activities that need to take into account the movement of objects in three-dimensional space.

[0127] It is understandable that the locations of multiple cameras should be set up to allow each camera to see every corner of the competition venue, and when processing data collected by multiple cameras, attention should also be paid to data synchronization and calibration. Since different cameras may have time differences or spatial differences, the collected data needs to be accurately synchronized and calibrated in time frame rate to ensure that the virtual objects created in the virtual scene can accurately reflect the position and posture of the actual participants.

[0128] 302, analyzing the video data frame by frame to determine the target image coordinates of the target object at each moment in the image coordinate system;

[0129] 303, based on the calibration parameters of the shooting device, convert the target image coordinates into target world coordinates in the world coordinate system;

[0130] 304, determining the position information based on the target world coordinates at each moment.

[0131] It can be understood that the above steps 302 to 304 correspond to Figure 2 Step 202 in the illustrated embodiment.

[0132] In this embodiment, for step 302, after the video data is acquired, the video data needs to be decoded and decomposed into separate image frames, each of which is a static image representing a time point in the video. In each frame of the image, a target detection algorithm is used to identify the target object. This can be done by using a pre-trained model (such as a deep learning model) that has been trained to recognize specific object categories. Once the target object is detected in a frame, the system can extract features of the object (such as edges, corners, color histograms, etc.). These features are then used to follow the target object between consecutive frames; for the target object detected in each frame, its coordinates in the image coordinate system are determined. This usually involves converting the bounding box or center point of the target object into pixel coordinates. These coordinates represent the position and size of the target object in the image. The image coordinates of the target object in each frame are recorded, and these coordinate data can be saved as records in a file or database for subsequent analysis or visualization. While recording the coordinates, the timestamp corresponding to each coordinate also needs to be recorded, which ensures the accurate correspondence between the coordinate data and the time information, so that the subsequent analysis can accurately understand the position of the target object at different times. This step corresponds to Figure 4 Multi-object detection network and multi-object tracking in.

[0133] In one possible line of sight method, step 302 specifically includes:

[0134] 3021, analyzing the video data frame by frame to determine the target image coordinates of the target object in each frame image;

[0135] 3022, based on the frame rate of the video data, calculate the target image coordinates of the target object at each moment.

[0136] It is understandable that to determine the target image coordinates of the target object at each moment, the target image coordinates of the target object in each frame image can be determined first, and then frame rate conversion is performed according to the frame rate of the video data to obtain the target coordinate image at each moment.

[0137] In a possible line of sight method, before step 302, the method further includes:

[0138] Perform moving target detection on the video data and determine the moving object in the video data as the target object.

[0139] It is understandable that in the embodiment, an algorithm for moving target detection is used. This detection is usually implemented based on methods such as inter-frame differences, background modeling, optical flow or deep learning, and usually includes: first, necessary preprocessing of the video data, including denoising, color space conversion (such as conversion from RGB to grayscale) and possible size adjustment, for subsequent processing and analysis. Then, in order to separate the moving object from the video, it is necessary to first establish a background model, which can be done by analyzing the initial frame or a series of frames of the video sequence to identify and model static background elements. During video playback, the background model needs to be continuously updated to adapt to possible background changes (such as lighting changes, dynamic background objects, etc.); then, using the background model, the current frame is compared with the background model to identify the difference between the foreground (i.e., the moving object) and the background; based on the foreground segmentation, the moving target object is further extracted and determined by connected region analysis, morphological operations or machine learning algorithms; for the detected moving targets, a target tracking algorithm can be applied to track them across multiple frames.

[0140] Specifically, there may be multiple target objects in the video data. Here, a multi-target tracking algorithm can be used to track the relevant objects detected in each video stream and obtain the image coordinates of each tracked object at each moment. For the tracked object P, it can be expressed as ,in, is the video stream number, is the id of the tracking object, is time, the pixel coordinates of each tracked object can be obtained by To express.

[0141] Regarding step 303 , after the target image coordinates of the target object are determined, the image coordinates may be converted into world coordinates in a world coordinate system based on calibration parameters of the photographing device.

[0142] For ease of understanding, the concepts involved in the embodiments of the present application are defined below:

[0143] Pixel coordinate system: Figure 5a The coordinate system shown is based on the upper left corner of the image and the unit is pixels;

[0144] Image coordinate system: Figure 5b The coordinate system shown takes the image origin as the midpoint of the imaging plane;

[0145] Camera coordinate system: Figure 5cThe coordinate system shown in the figure takes the optical center of the camera as the origin and the z-axis along the focal length direction as the coordinate system shown in the figure;

[0146] World coordinate system: A suitable real-world coordinate system selected based on the event venue;

[0147] Projection matrix: describes the relationship between the camera's 2D image coordinate points and the 3D object coordinate points.

[0148] Next, step 303 in the embodiment of the present application is described in detail:

[0149] a) Due to the transformation relationship between the camera and the world coordinates, the camera coordinates of an object at a certain moment in a video stream With world coordinates The relationship is as Figure 6a As shown:

[0150] The expression for the change from the world coordinate system to the camera coordinate system is:

[0151] ;

[0152] in is the rotation matrix from world coordinates to camera coordinates, is the translation matrix from world coordinates to camera coordinates, which is and are all known quantities.

[0153] b) Due to the pinhole imaging principle, the relationship between the image coordinates and the camera coordinates is as follows Figure 6b As shown, we can deduce from similar triangles:

[0154] ;

[0155] Where f is the focal length of the camera, which is a known quantity;

[0156] c) If Figure 6c As shown, the image coordinate system To pixel coordinate system The relationship can be obtained:

[0157] ;

[0158] in , is the scaling factor from pixel units to length units, , is the initial offset, which are the intrinsic parameters of the camera and can be obtained by checking the camera parameters or performing camera calibration;

[0159] d) Based on the above, the relationship between the pixel coordinates of an object at a certain moment in a video stream and the world coordinates is converted to:

[0160] ;

[0161] Simplifying all variables gives:

[0162] ;

[0163] in is the camera's projection matrix, : 3×4.

[0164] In one possible case, when the video data is treated as two-dimensional, that is, , a single camera can obtain the true two-dimensional coordinates of an object.

[0165] For step 304, after obtaining the target world coordinates of the target object at each moment, the position and posture information of the target object can be determined. This step involves the accurate calculation of the position and posture of the object in three-dimensional space. The world coordinates usually represent the position of the object in the global coordinate system, and the position and posture information includes the position and posture of the object, that is, the orientation and position of the object in space.

[0166] First, if Figure 6d As shown in the figure, by tracking the same target object in multiple video streams, the real coordinates of the target object at each moment can be solved. For a certain moment t and a certain object j, there is the following equation:

[0167] ;

[0168] The real coordinates of an object at each moment can be obtained by the least squares method: , recorded as .

[0169] Then, by taking the derivative with respect to time t, we can get the speed of any object related to the event at any time: , acceleration :

[0170] ;

[0171] In summary, we can obtain the relevant items related to the event, the position, speed, acceleration information of the athletes at each moment, that is, the posture information. The above steps correspond to Figure 4 The spatial pose in is settled to obtain the position, velocity and acceleration information.

[0172] In a possible implementation method, the installation position of the camera is unknown, so to convert the image coordinate system into the world coordinate system, the calibration parameters of the camera must be calculated first. Therefore, before step 303, the following is also included:

[0173] Obtaining the fixed world coordinates of the fixed object in the video data in the world coordinate system;

[0174] Determine the fixed image coordinates of the fixed object in the image coordinate system through the video data;

[0175] By fixing the world coordinates and the image coordinates, the calibration parameters are calculated.

[0176] Among them, fixed objects refer to objects that do not move in the video data, such as tables and chairs in the video, and field markings in standard competition venues. Taking field markings as an example, the sizes of each marking in a standard competition venue are consistent, so the size of each marking in the image coordinate system is compared with the actual size to calculate the calibration parameters of the camera.

[0177] It can be understood that, through the calculation in 303 above, the projection matrix P of the camera can be expressed as:

[0178] ;

[0179] In other words, as long as we know the fixed positions of n (n ≥ 4) points in the world coordinate system in the image coordinate system, we can solve the equations for P.

[0180] 305, creating a virtual scene based on the fixed objects in the video data;

[0181] 306, creating a virtual object based on the target object;

[0182] 307, determining the position and form of the virtual object at each moment in the virtual scene based on the posture information;

[0183] 308, based on the position form, setting the virtual object in the virtual scene.

[0184] It can be understood that the above steps 305 to 308 correspond to Figure 2 Step 203 in the illustrated embodiment.

[0185] In this embodiment, after determining the position information of each target object in the world coordinate system, the environment in the entire video data can be reconstructed by using a virtual simulator (such as Unral Unreal Engine), including the corresponding virtual scene and virtual objects. When the video data is a competition video of a competitive game, the competition environment of the entire game can be reconstructed. The process is as follows:

[0186] Create a race map in the simulator (if conditions permit, add the same texture information as in reality to the map): First, you need to design and build a race map in the simulation software. This map should reflect the real-world race environment as much as possible, including track layout, obstacles, spectator seats, etc. If conditions permit, add the same texture information as in reality to the map, including using high-resolution texture maps to simulate the material, color, and lighting effects of the ground to make the simulation map look more realistic.

[0187] Create corresponding characters, objects, etc. in the simulator (if conditions permit, add the same texture information as the display): Next, create the characters and objects required for the event in the simulation environment. These can include contestants, spectators, referees, racing cars, obstacles, etc. Similarly, if conditions permit, add the same texture information as in reality to these characters and objects, such as adding real body maps to racing cars, adding real clothing textures to characters, etc. This not only improves the visual effect, but also increases the authenticity of the simulation environment.

[0188] Through the above-obtained position information f(j, t), velocity information v(j, t), acceleration information a(j, t); to set the characters and objects in the simulation environment, so that the simulator simulates each frame the same as the real situation: During the simulation process, by real-time acquisition and processing of position information f(j, t), velocity information v(j, t) and acceleration information a(j, t), the characters and objects in the simulation environment can be accurately set and adjusted. This information can be obtained through the physical engine or sensor model of the simulation software. Based on this information, the motion trajectory, velocity change, acceleration distribution and other parameters of the simulation object can be adjusted to make the simulator simulate each frame the same as the real situation.

[0189] 309, creating a virtual character in the virtual scene in response to the character adding request;

[0190] 310, adding a character perspective to the virtual character, and returning a picture of the virtual scene from the character perspective.

[0191] In this embodiment, the user is allowed to add a virtual character and obtain the perspective information of a specific character.

[0192] Here is a detailed explanation of these two steps:

[0193] When a user wants to add a virtual character, the simulator also adds the corresponding virtual character in the virtual environment to interact with the virtual environment: In the simulation environment, users may want to add their own virtual characters and flexibly adjust the viewing angle by operating the virtual characters to move in the virtual scene. These virtual characters can have customized appearances. The user's character addition request can be obtained through the corresponding interface. When the user adds a virtual character through the interface, the simulator will dynamically generate the corresponding virtual entity in the virtual environment. In addition, a control interface for the virtual character can also be provided, allowing the user to manipulate the movement and behavior of the virtual character through the keyboard, mouse, game controller or other input devices. In this way, users can personally participate in the simulation environment and experience real-time interaction with the virtual character.

[0194] When a user wants to obtain the perspective of a certain character, the corresponding camera information is returned to the user: In the simulation environment, the user may want to observe the event or other activities from the perspective of a specific character. To achieve this function, the simulator can pre-configure a virtual camera for each key character or object. When the user requests to obtain the perspective of a certain character, the simulator activates the camera associated with the character and captures the real-time picture from the perspective of the camera. The simulator can then return the captured camera information to the user in the form of a video stream. This information can be transmitted to the user's client device via the network and displayed on the user interface. Users can experience the perspective of a specific character by watching these video streams and better understand the dynamics and details in the simulation environment. In addition, the simulator can also provide the function of perspective switching, allowing users to switch freely between multiple characters or cameras. In this way, users can flexibly adjust the observation perspective as needed and obtain information from different angles and fields of view.

[0195] Furthermore, before creating the virtual character, the method further includes: creating interactive elements in the virtual scene, where the interactive elements are used to respond to interactive operations of the virtual character and interact with the virtual character.

[0196] It is understandable that creating diverse interactive elements in virtual scenes can further enhance the user's interactivity and sense of participation. These interactive elements not only enrich the details of the virtual environment, but also provide more interactive opportunities, allowing users to interact more deeply with the virtual world.

[0197] Among them, the interactive element can be a chair, which provides users with an interactive experience of sitting down to rest or watching the game. The user can operate the virtual character to walk to the chair and trigger the action of sitting down. The interactive element can also be props such as salutes and whistles, which can increase the interactive fun of users in the virtual scene. When the user operates the virtual character to use the salute, the explosion effect of the salute can be triggered, adding a festive or celebratory atmosphere to the virtual scene. The whistle can be used to make sounds or as a command signal to interact with other virtual characters or scene elements. The use of these props can not only enhance the user's sense of participation, but also add more dynamics and changes to the virtual scene.

[0198] 311, adding a virtual perspective to a virtual object;

[0199] 312 , in response to the perspective selection operation on the target object, displaying a picture of the virtual scene under the virtual perspective.

[0200] It can be understood that the above steps 311 and 312 correspond to Figure 2 Step 204 and step 205 in the illustrated embodiment.

[0201] In this embodiment, the virtual object can be a key task or item. By adding a virtual camera component to the key person or item, it is possible to view the image in the virtual scene from the perspective of the virtual object. Specifically, by adding cameras to the target object, these cameras can simulate the perspective and parameters of cameras in the real world, thereby capturing real-time images in the simulation environment. By adjusting the position, angle, focal length and other parameters of the camera, video information with different perspectives and detail levels can be obtained.

[0202] In a possible implementation method, the method further includes:

[0203] In response to the viewing angle adjustment operation, the viewing angle of the picture is changed; the viewing angle adjustment operation includes any one of a viewing angle translation operation, a viewing angle rotation operation, and a viewing angle zoom operation.

[0204] It is understandable that after the user determines the viewing angle, the image under that viewing angle will be displayed to the user. In order to allow the user to browse and interact with the content in the virtual scene more flexibly and to enhance the user experience and immersion, a viewing angle adjustment function can be provided. The user can perform viewing angle adjustment operations, which mainly include the following three operations:

[0205] Perspective panning operation: allows users to move the perspective of the screen horizontally or vertically to observe content in different locations. For example, in a virtual scene, users can view scenes and objects in different areas by panning the perspective. This operation is usually achieved by dragging the mouse, touching the screen, or using specific control keys.

[0206] Perspective rotation operation: allows the user to change the direction of the perspective so that the scene or object can be viewed from different angles. Rotating the perspective can be achieved by rotating the mouse wheel, using specific keyboard shortcuts, or through touch gestures.

[0207] Zooming: Allows the user to adjust the zoom level of the image, thereby changing the focal length of the view. Zooming can usually be achieved by rolling the mouse wheel, using the zoom shortcut keys on the keyboard, or by touching the zoom control on the screen.

[0208] These viewing angle adjustment operations can be used individually or in combination to achieve precise control of the viewing angle of the picture.

[0209] For ease of understanding, a video data processing method applied to event videos is introduced below.

[0210] See also Figure 7 , Figure 7 A product flow chart of the video data processing method provided in an embodiment of the present application.

[0211] Multiple fixed cameras are used to record various angles of the game and obtain multiple video streams. After performing target detection, multi-target tracking, pose calculation, scene reconstruction and other processing based on multiple video streams, a virtually reconstructed event scene can be obtained. The tasks and objects in the scene are consistent with those in reality. Users can get the following experience through the software and the virtual environment.

[0212] 1) Obtain a global picture from a bird's eye view during the game to help users conduct a more comprehensive formation analysis;

[0213] 2) Obtain the perspective information of any athlete to help users experience the game from the perspective of the player;

[0214] 3) Obtaining perspective information of virtual objects in the game, such as footballs, goals, etc.;

[0215] 4) The avatar image is placed in the virtual environment to experience the competition immersively.

[0216] like Figures 8a to 8d As shown, Figures 8a to 8d This is a practical application diagram of the video data processing method provided in the embodiment of the present application.

[0217] like Figure 8a As shown, this method is tested on a football match to obtain a recorded match video, which includes multiple moving objects such as athletes, referees, spectators, and fixed objects such as lawns, field lines, and goals.

[0218] like Figure 8bAs shown, the target detection and multi-target tracking algorithms are used to track the athletes and referees in the video.

[0219] like Figure 8c As shown, through coordinate solution, a virtual scene is constructed based on the position information of each athlete and referee, and a reconstructed global view (top view) can be obtained based on the virtual scene.

[0220] like Figure 8d As shown, when a user wants to obtain a virtual perspective of a certain athlete, in response to the user's selection operation on the athlete, the game is watched from the athlete's perspective.

[0221] Through the above process, it is possible to simulate and obtain perspective information that is not in the original video, and to add virtual characters to interact with it, so as to obtain the actual situation of the game from any perspective.

[0222] The video data processing device in this application is described in detail below. Fig. 9 . Fig. 9 This is a schematic diagram of an embodiment of a video data processing device 900 in an embodiment of the present application. The video data processing device 900 includes:

[0223] An acquisition module 901 is used to acquire video data of a competitive game;

[0224] A posture calculation module 902 is used to track the target in the video data and determine the posture information of the target object at each moment;

[0225] A modeling module 903, for creating a virtual object corresponding to the target object in a virtual scene based on the posture information, the virtual scene being created based on a competition venue;

[0226] A perspective adding module 904, used to add a virtual perspective to a virtual object;

[0227] The display module 905 is used to display the image of the virtual scene under the virtual perspective in response to the perspective selection operation on the target object.

[0228] In a possible implementation method, the acquisition module 901 is specifically used to acquire video data collected by a fixed shooting device;

[0229] The posture calculation module 902 is specifically used to determine the target image coordinates of the target object at each moment in the image coordinate system; based on the calibration parameters of the shooting device, the target image coordinates are converted into the target world coordinates in the world coordinate system; based on the target world coordinates at each moment, the posture information is determined.

[0230] In a possible implementation method, the method further includes:

[0231] The target detection module is used to perform moving target detection on the video data and determine the moving object in the video data as the target object.

[0232] In a possible implementation method, the method further includes:

[0233] The calibration calculation module is used to obtain the fixed world coordinates of the fixed object in the video data in the world coordinate system; determine the fixed image coordinates of the fixed object in the image coordinate system through the video data; and calculate the calibration parameters through the fixed world coordinates and the fixed image coordinates.

[0234] In a possible implementation method, the posture calculation module 902 is specifically used to analyze the video data frame by frame to determine the target image coordinates of the target object in each frame image; based on the frame rate of the video data, the target image coordinates of the target object at each moment are calculated.

[0235] In a possible implementation method, the acquisition module 901 is specifically used to acquire video data collected by multiple shooting devices at different positions; the posture information is specifically posture information in a three-dimensional coordinate system.

[0236] In one possible implementation method, the modeling module 903 is specifically used to create a virtual scene based on fixed objects in video data; create a virtual object based on the target object; determine the position form of the virtual object at each moment in the virtual scene based on posture information; and set the virtual object in the virtual scene based on the position form.

[0237] In a possible implementation method, the method further includes:

[0238] The role adding module is used to create a virtual role in a virtual scene in response to a role adding request; add a role perspective to the virtual role, and return a picture of the virtual scene under the role perspective.

[0239] In a possible implementation method, the method further includes:

[0240] The element adding module is used to create interactive elements in the virtual scene, and the interactive elements are used to respond to the interactive operations of the virtual characters and interact with the virtual characters.

[0241] In a possible implementation method, the virtual scene and the virtual objects are created by Unreal Engine.

[0242] In a possible implementation method, the method further includes:

[0243] The viewing angle adjustment module is used to change the viewing angle of the picture in response to the viewing angle adjustment operation; the viewing angle adjustment operation includes the viewing angle translation operation, the viewing angle rotation operation and the viewing angle zoom operation.

[0244] It can be understood that the video data processing device provided in the embodiment of the present application is used to perform the above Figure 2 and Figure 3 For the detailed description of the method in the corresponding embodiment, please refer to the above text, which will not be repeated here.

[0245] Fig.10 : is a schematic diagram of a server structure provided by an embodiment of the present application. The server 300 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 322 (for example, one or more processors) and memory 332, and one or more storage media 330 (for example, one or more mass storage devices) storing application programs 342 or data 344. Among them, the memory 332 and the storage medium 330 can be short-term storage or permanent storage. The program stored in the storage medium 330 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the server. Furthermore, the central processing unit 322 can be configured to communicate with the storage medium 330 to execute a series of instruction operations in the storage medium 330 on the server 300.

[0246] The server 300 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input and output interfaces 358, and / or one or more operating systems 341, such as Windows Server 2000. TM , Mac OS X TM , Unix TM , Linux TM , FreeBSD TM etc.

[0247] The steps performed by the server in the above embodiment can be based on the Fig.10 The server structure shown.

[0248] It is understandable that in the specific implementation of the present application, related data such as video data is involved. When the above embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0249] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0250] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0251] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0252] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0253] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0254] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0255] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some of the technical features therein by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A video data processing method, characterized in that: include: Obtain video data of competitive games; Performing moving target detection on the video data, and determining a moving object in the video data as a target object; Performing target tracking on the video data to determine the position and posture information of the target object at each moment; Based on the position information, creating a virtual object corresponding to the target object in a virtual scene, wherein the virtual scene is created based on a competition venue of the competitive game; Adding a virtual perspective to the virtual object corresponding to the target object; The virtual object moves synchronously according to the position information of the target object at each moment to simulate the actual game situation; the virtual perspective represents the perspective of the target object; The virtual object corresponding to the target object includes a virtual item; creating interactive elements in the virtual scene; In response to a character adding request, a virtual character is created in the virtual scene; the interactive element is used to interact with the virtual character in response to an interactive operation of the virtual character; Adding a character perspective to the virtual character, and returning the image of the virtual scene from the character perspective, so that the user can adjust the character perspective by operating the virtual character to move in the virtual scene; In response to a perspective selection operation on the target object, displaying a picture of the virtual scene under the virtual perspective, so that a user watches the competitive game from the perspective of the target object; In response to a perspective adjustment operation, the perspective of the screen is changed so that the user watches the competitive game from different perspectives of the target object; the perspective adjustment operation includes a perspective translation operation, a perspective rotation operation, and a perspective zoom operation.

2. The method according to claim 1, characterized in that The step of obtaining video data of a competitive game includes: Acquiring the video data collected by a fixed shooting device; The performing target tracking on the video data to determine the position information of the target object at each moment includes: Analyze the video data frame by frame to determine the target image coordinates of the target object at each moment in the image coordinate system; Based on the calibration parameters of the photographing device, the target image coordinates are converted into target world coordinates in a world coordinate system; The position information is determined based on the target world coordinates at each moment.

3. The method according to claim 2, characterized in that Before converting the target image coordinates into target world coordinates in a world coordinate system based on the calibration parameters of the shooting device, the method further includes: Obtaining fixed world coordinates of a fixed object in the video data in the world coordinate system; Determining fixed image coordinates of the fixed object in the image coordinate system through the video data; The calibration parameters are calculated using the fixed world coordinates and the fixed image coordinates.

4. The method according to claim 2, characterized in that: The step of analyzing the video data frame by frame to determine the target image coordinates of the target object at each moment in the image coordinate system includes: Analyze the video data frame by frame to determine the target image coordinates of the target object in each frame image; The target image coordinates of the target object at each moment are calculated based on the frame rate of the video data.

5. The method according to claim 1, characterized in that The step of obtaining video data of a competitive game includes: The video data collected by multiple shooting devices at different positions is obtained; the posture information is specifically the posture information in a three-dimensional coordinate system.

6. The method according to claim 1, characterized in that The step of creating a virtual object corresponding to the target object in a virtual scene based on the posture information includes: Creating the virtual scene based on the fixed objects in the video data; creating the virtual object based on the target object; Based on the posture information, determining the position form of the virtual object at each moment in the virtual scene; Based on the position form, the virtual object is set in the virtual scene.

7. The method according to claim 1, characterized in that The virtual scene and the virtual object are created by Unreal Engine.

8. A video data processing device, characterized in that: include: An acquisition module, used to acquire video data of competitive games; A target detection module, used to perform moving target detection on the video data, and determine the moving object in the video data as a target object; A posture calculation module is used to track the target on the video data and determine the posture information of the target object at each moment; A modeling module, configured to create a virtual object corresponding to the target object in a virtual scene based on the posture information, wherein the virtual scene is created based on a competition venue of the competition; A perspective adding module, used to add a virtual perspective to the virtual object corresponding to the target object; The virtual object moves synchronously according to the position information of the target object at each moment to simulate the actual game situation; the virtual perspective represents the perspective of the target object; The virtual object corresponding to the target object includes a virtual item; An element adding module, used for creating interactive elements in the virtual scene, wherein the interactive elements are used for interacting with the virtual character in response to the interactive operation of the virtual character; A character adding module, for creating a virtual character in the virtual scene in response to a character adding request; adding a character perspective to the virtual character, and returning a picture of the virtual scene under the character perspective, so that a user can adjust the character perspective by operating the virtual character to move in the virtual scene; A display module, configured to display the image of the virtual scene under the virtual perspective in response to the perspective selection operation of the target object, so that the user can watch the competitive game from the perspective of the target object; The perspective adjustment module is used to change the perspective of the picture in response to a perspective adjustment operation, so that the user can watch the competitive game from different perspectives of the target object; the perspective adjustment operation includes a perspective translation operation, a perspective rotation operation and a perspective zoom operation.

9. The device according to claim 8, characterized in that The acquisition module is specifically used for: Acquiring the video data collected by a fixed shooting device; The posture calculation module is specifically used to analyze the video data frame by frame to determine the target image coordinates of the target object at each moment in the image coordinate system; based on the calibration parameters of the shooting device, convert the target image coordinates into target world coordinates in the world coordinate system; based on the target world coordinates at each moment, determine the posture information.

10. The device according to claim 9, characterized in that Also includes: A calibration calculation module, used for obtaining fixed world coordinates of a fixed object in the video data in the world coordinate system; The fixed image coordinates of the fixed object in the image coordinate system are determined by the video data; and the calibration parameters are calculated by the fixed world coordinates and the fixed image coordinates.

11. The device according to claim 9, characterized in that The posture calculation module is specifically used to analyze the video data frame by frame to determine the target image coordinates of the target object in each frame image; based on the frame rate of the video data, calculate the target image coordinates of the target object at each moment.

12. The device according to claim 8, characterized in that The acquisition module is specifically used to acquire the video data collected by multiple shooting devices at different positions; the posture information is specifically posture information in a three-dimensional coordinate system.

13. The device according to claim 8, characterized in that The modeling module is specifically used to create the virtual scene based on the fixed objects in the video data; The virtual object is created based on the target object; the position form of the virtual object at each moment in the virtual scene is determined based on the posture information; and the virtual object is set in the virtual scene based on the position form.

14. The device according to claim 8, characterized in that The virtual scene and the virtual object are created by Unreal Engine.

15. A computer device, characterized in that: include: memory, and processor; The memory stores instructions, and when the instructions are executed on the processor, the video data processing method according to any one of claims 1 to 7 is implemented. 16 . A computer-readable storage medium comprising instructions, which, when executed on a computer, enables the computer to execute the video data processing method according to claim 1 .

17. A computer program product comprising a computer program, characterized in that The computer program is used by a processor to execute the video data processing method according to any one of claims 1 to 7.

Images