Video display method and device, storage medium and program product

Abstract

The invention discloses a video display method and device, a storage medium and a program product, and relates to the technical field of communication, in order to improve watching experience of a user on live broadcast of an event, the method comprises the following steps: acquiring an image of the live broadcast event by using a plurality of image acquisition devices arranged at a site of the live broadcast event; positioning and tracking the target person and / or the competition ball in the image to obtain first information; wherein the first information comprises three-dimensional model driving parameters of the target person and / or three-dimensional position information of the competition ball; the first information and the audio and video data of the live broadcast event are sent to a terminal, so that the terminal generates a three-dimensional virtual live broadcast picture of the live broadcast event based on the first information and the audio and video data and displays the three-dimensional virtual live broadcast picture, and the watching experience of a user on live broadcast of the event can be improved.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to a video display method, device, storage medium and program product. Background Art

[0002] With the development of live sports broadcasting, the current mainstream immersive live broadcasting system with switchable viewing angles is generally implemented through multi-channel camera live broadcasting. However, in the existing technology, the viewing angle of users when watching live sports broadcasting is limited to the position of the camera equipment, and it is impossible to switch any viewing angle, which affects the user's viewing experience. Summary of the invention

[0003] The embodiments of the present application provide a display method, device, storage medium and program product to improve the user's viewing experience of live events.

[0004] In a first aspect, an embodiment of the present application provides a video display method, which is applied to a server, comprising:

[0005] Using a plurality of image acquisition devices disposed at a venue where a live event is broadcast, images of the live event are acquired;

[0006] Positioning and tracking the target person and / or the match ball in the image respectively to obtain first information; wherein the first information includes the three-dimensional model driving parameters of the target person and / or the three-dimensional position information of the match ball;

[0007] The first information and the audio and video data of the live event are sent to a terminal, so that the terminal generates a three-dimensional virtual live screen of the live event based on the first information and the audio and video data and displays the three-dimensional virtual live screen.

[0008] In a second aspect, the embodiment of the present application further provides a video display method, which is applied to a client, comprising:

[0009] Acquire first information sent by a server and audio and video data of a live event at a target time; wherein the first information includes three-dimensional model driving parameters of a target person in the live event and / or three-dimensional position information of a match ball; the server is used to execute the video display method executed by the server;

[0010] Obtaining a three-dimensional virtual live screen of the live event according to the first information and the audio and video data;

[0011] The three-dimensional virtual live broadcast screen is displayed.

[0012] In a third aspect, an embodiment of the present application further provides a communication device, comprising: a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor implements the steps in the video display method described above when executing the program.

[0013] In a fourth aspect, an embodiment of the present application further provides a readable storage medium, on which a program is stored, and when the program is executed by a processor, the steps in the video display method as described above are implemented.

[0014] In a fifth aspect, an embodiment of the present application further provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps in the video display method as described above.

[0015] In the embodiment of the present application, the server obtains images of the live event at different times, locates and tracks the target person and / or the match ball in the image, obtains first information, and sends the first information and audio and video data at different times to the terminal, so that the terminal can use the first information and the audio and video data to obtain the three-dimensional virtual live screen of the live event and display the three-dimensional virtual live screen. Therefore, the terminal can switch the viewing angle at will, thereby improving the user's viewing experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is one of the flow charts of the video display method provided in the embodiment of the present application;

[0017] Figure 2 is a schematic diagram of a coordinate system of an embodiment of the present application;

[0018] Figure 3 is a schematic diagram of obtaining an image subset in an embodiment of the present application;

[0019] Figure 4 is a schematic diagram of positioning and tracking a game ball in an embodiment of the present application;

[0020] Figure 5 This is the second flowchart of the video display method provided in the embodiment of the present application;

[0021] Figure 6 is one of the structural diagrams of the video display device provided in the embodiment of the present application;

[0022] Figure 7 This is the second structural diagram of the video display device provided in the embodiment of the present application;

[0023] Figure 8 It is a structural diagram of the communication device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] In the embodiments of the present application, the term "and / or" describes the association relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0025] In the embodiments of the present application, the term "plurality" refers to two or more than two, and other quantifiers are similar.

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] See also Figure 1 , Figure 1 is a flow chart of a video display method provided in an embodiment of the present application, which is applied to a server, such as Figure 1 As shown, the following steps are included:

[0028] Step 101: Utilize a plurality of image acquisition devices set up at a venue where a live event is being broadcast to acquire images of the live event.

[0029] In the embodiment of the present application, multiple image acquisition devices may be arranged around the venue according to factors such as the field of view (FOV) of the image acquisition device (such as an RGBD camera) and the size of the venue for the live event. Multiple image acquisition devices acquire images at multiple identical moments. For example, multiple image acquisition devices acquire images at moment t0, and multiple image acquisition devices acquire images at moment t1.

[0030] In the process of setting the position of the image acquisition device, the viewing angles of two adjacent image acquisition devices can have at least 20% overlap to ensure full coverage of the entire venue. Each image acquisition device is connected to the server for frame synchronization to ensure that images are taken at the same time. Afterwards, the image acquisition device can be calibrated to obtain the internal and external parameters of each image acquisition device. The internal parameters may include resolution, focal length, etc.; the external parameters may include the position of the image acquisition device, such as the position and orientation of the image acquisition device in three-dimensional space. Finally, the actual position of the external parameters of each image acquisition device in the venue is unified.

[0031] like Figure 2As shown, taking a basketball game as an example, the point W0 shown in the figure can be used as the coordinate origin to establish a reference coordinate system. Subsequent processes, such as calculating the position of the human body and the position of the game ball, all use this coordinate system as the reference coordinate system. Among them, the point W0 is any one of the four vertices of the game field. Of course, in actual applications, other points of the game field can also be selected as the coordinate origin.

[0032] In the embodiment of the present application, a virtual digital human with a high degree of realism can also be generated to improve the viewing experience. For example, full-body photos, height and weight, high-definition facial photos and other information of all athletes can be collected to generate a human body geometry model that conforms to the athlete's body and face, and has high-definition textures. After that, after fitting the human body geometry model with the standard model, the joints and face are automatically bound, and a digital human can be generated, and the digital human has the ability to be driven. Among them, one or all athletes can be used as the target person for subsequent description.

[0033] Before the game starts, the virtual environment of the competition venue is prefabricated through UE (Unreal Engine) engine, Unity engine, etc., including the competition ground, stands, basketball stands, billboards and other necessary items. Take basketball games as an example. Because the venues for basketball games are fixed, they can be built in advance according to the venue environment. Through the generated virtual environment, ground advertisements or surrounding billboard content can be changed at any time during the game as needed.

[0034] Step 102: Position and track the target person and / or the match ball in the image respectively to obtain first information; wherein the first information includes the three-dimensional model driving parameters of the target person and / or the three-dimensional position information of the match ball.

[0035] Among them, the image of the live event includes: the images of the live event collected by the multiple image acquisition devices at multiple identical times, each of the images of the live event includes: an RGB image and a depth image, and the three-dimensional model driving parameters of each target person include: Skinned Multi-PersonLinear Model (SMPL) parameters, scale parameters, and spatial position parameters.

[0036] Since there may be problems such as occlusion and motion blur in the competition venue, in the embodiment of the present application, multiple image acquisition devices and multiple characters are used for simultaneous tracking and posture estimation. Specifically, the target characters in the live event image are located and tracked, and the three-dimensional model driving parameters of each target character are obtained, including the following:

[0037] S1. Determine multiple image sets of a first target person based on images captured by multiple image acquisition devices at a first moment; wherein each image set includes: an RGB image, a depth image and a corresponding mask image of the first target person, the first moment is any moment during the live event, the first target person is any person among the target persons, and the first target person is any person in the live event, such as any athlete, etc.

[0038] The images captured by each image acquisition device at each moment include red, green, blue (RGB) images and depth images.

[0039] In this step, for each image taken by the image acquisition device at the first moment, human body detection is performed based on the RGB image and depth image in the image to generate a first bounding box; wherein the depth value of the human body in the first bounding box is less than or equal to a preset value.

[0040] Assume that an image acquisition device captures a set of RGB images f1 and depth images D1 at time t (the first moment). Using the You Only Look Once (YOLO) deep learning library, the human bodies in the RGB image f1 and the depth image D1 are detected to generate a bounding box for each human body. Here, in order to improve efficiency, the human bodies that are far away from the current image acquisition device can be eliminated. For example, the RGB image f1 and the depth image D1 can be aligned to extract the depth of the human body in each bounding box in the depth image D1. If the depth of a human body is greater than a preset value (which can be set according to actual needs), it means that the area occupied by the human body in the current depth image is small, and the bounding box can be abandoned.

[0041] After obtaining the above-mentioned first bounding box, the human body in the first bounding box is identified to determine whether the first bounding box includes the first person. If multiple first bounding boxes include the first target person, the RGB image is segmented to obtain a mask map. For example, segmentation technology can be used, such as calling the detectron2 segmentation library to segment the RGB image to obtain a mask map. Afterwards, multiple sets of the RGB images, the depth images and the mask maps are used to form multiple image sets of the first target person.

[0042] Specifically, according to the arrangement of the image acquisition devices, there is visual overlap between the image acquisition devices, so the same person can be detected simultaneously on different image acquisition devices, but the viewing angles are different. In order to establish a connection between the human bodies identified in different images, it is necessary to identify the identified human bodies. For example, the facial area of ​​the human body in the first bounding box can be identified to obtain the identity. If the face is blocked, the number on the back of the human body can be identified to obtain the athlete number, thereby determining the person in the first bounding box. The person can be one or more. If the recognition of a first bounding box fails multiple times, for example, the corresponding person is not identified, the bounding box can be abandoned.

[0043] After character recognition, the images of a certain character can be aggregated. For example, suppose image f1 recognizes athletes P1 and P2, image f2 recognizes athletes P3 and P4, image f3 recognizes athletes P1 and P3, and image f4 recognizes athletes P2 and P4. Then, the image set of athlete P1 is image f1 and image f3; the image set of athlete P2 is image f1 and image f4, and so on. At the same time, the depth images and mask images corresponding to images f1 and f3 are combined to obtain each image subset. Using multiple image subsets, the final image set can be formed.

[0044] S2. Obtain multiple SMPL parameters according to the RGB images and mask images in the multiple image sets.

[0045] S3. Obtaining multiple scale parameters according to the depth images and RGB images in the multiple image sets;

[0046] S4. Obtaining a plurality of spatial position parameters according to the depth images in the plurality of image sets;

[0047] S5. Determine the three-dimensional model driving parameters of the first target person at the first moment according to the multiple skinned multi-person linear model SMPL parameters, the multiple scale parameters, and the multiple spatial position parameters.

[0048] Here, the first moment is taken as an example for description. Each first moment has corresponding SMPL parameters, scale parameters and spatial position parameters.

[0049] Among them, SMPL is a computer graphics model for modeling human body shape and posture. It is a parametric model for generating a three-dimensional human body model, allowing researchers to adjust the shape and posture of the human body according to parameters. It includes body shape parameters, body posture parameters, global posture parameters, expression parameters, finger parameters, etc. Here, the SMPL parameters of the 3D human body, that is, the SMPL parameters, can be regressed from the RGB image and mask image at the first moment based on, for example, the Regularized Orthogonal Matching Pursuit (ROMP) algorithm. Afterwards, the scale parameters can be obtained based on the depth image and the RGB image to solve the scale problem of the SMPL model.

[0050] For example, taking athlete P1 as an example, here, the depth information of the human body is extracted from the depth map closest to athlete P1, and the depth information is converted into a three-dimensional point cloud in the world coordinate system according to the intrinsic and extrinsic parameters of the image acquisition device. The initially generated SMPL model is aligned with the three-dimensional point cloud, and the scale of the SMPL model is calculated by minimizing the error to obtain the scale parameter. For example, the iterative closest point (ICP) algorithm can be used to iteratively adjust the scale and position of the SMPL model so that it is optimally aligned with the three-dimensional point cloud.

[0051] The spatial position parameter at the first moment is based on the position of the first target person relative to the image acquisition device at that moment and the position of the image acquisition device relative to the aforementioned Figure 2 It can also be understood that the spatial position parameter is the spatial position parameter of the first target person relative to the basic coordinate system. Still taking the athlete P1 as an example, since the position of the athlete P1 relative to the image acquisition device is available, and the image acquisition device relative to the aforementioned Figure 2 The position of the coordinate system shown is also known, therefore, the spatial position parameters of the athlete relative to the coordinate system can be obtained.

[0052] refer to Figure 3As shown, it is a schematic diagram of obtaining an image subset. Detection, image segmentation, and recognition are performed on the RGB image f1 and the depth image D1, and the characters included are P1 and P2; detection, image segmentation, and recognition are performed on the RGB image f2 and the depth image D2, and the characters included are P3 and P4; detection, image segmentation, and recognition are performed on the RGB image f3 and the depth image D3, and the characters included are P1 and P3; detection, image segmentation, and recognition are performed on the RGB image f4 and the depth image D4, and the characters included are P2 and P4. The image subsets formed are: P1: f1, f3, D1, D3, m1, m3; P2: f1, f4, D1, D4, m1, m4; P3: f2, f3, D2, D3, m2, m3; P4: f2, f4, D2, D4, m2, m4.

[0053] In the above step S5, the SMPL parameter mean, the scale parameter mean and the spatial position parameter mean may be determined according to the multiple skinned multi-person linear model SMPL parameters, the multiple scale parameters and the multiple spatial position parameters.

[0054] Next, the preset three-dimensional human body model corresponding to the first target person is driven by using the mean value of the SMPL parameter, the mean value of the scale parameter and the mean value of the spatial position parameter to obtain a candidate three-dimensional human body model.

[0055] Then, the candidate three-dimensional human body model is projected into a two-dimensional image plane to obtain a two-dimensional contour of the candidate three-dimensional human body model.

[0056] Specifically, in the process of three-dimensional projection, the candidate three-dimensional human body model in the three-dimensional world can be mapped onto a two-dimensional plane. For each image acquisition device viewing angle, the vertices of the three-dimensional human body model can be projected onto the image plane using the intrinsic matrix and extrinsic matrix of the image acquisition device to form a two-dimensional outline.

[0057] Afterwards, the two-dimensional contour is compared with the actual contour of the first target person in the RGB image to obtain a comparison result. Specifically, the edge or contour loss of the two-dimensional contour and the actual contour can be calculated.

[0058] Finally, the SMPL parameter mean, the scale parameter mean, and the spatial position parameter mean are adjusted according to the comparison result to obtain the three-dimensional model driving parameters of the first target person at the first moment.

[0059] Specifically, deep learning back propagation and gradient descent methods can be used to adjust the SMPL parameter mean, scale parameter mean and spatial position parameter mean to obtain the three-dimensional model driving parameters of the first character at the first moment to reduce image loss.

[0060] For example, one or more means may be adjusted according to the comparison result, and the above process may be repeated using the adjusted parameters to obtain a new comparison result. If the new comparison combination meets expectations, the adjustment of the parameters is stopped, otherwise the above process may be continued or the adjustment end condition is met.

[0061] Combination Figure 4 As shown, for positioning and tracking the match ball, first traverse all images taken at the first moment, and identify the match ball in all images. Afterwards, the images that successfully identify the match ball form an image set, and a mask image of the match ball is generated for each image in the image set. Then, based on the obtained mask image, the feature points of the match ball are extracted, and feature point matching and sparse three-dimensional reconstruction are performed. At the same time, a preset algorithm (such as bundle adjustment) can be used to optimize the spatial position to obtain the position of the center point of the match ball, that is, the three-dimensional position information of the match ball. In the above processing, for the frame at time t, if the position information of the match ball at time t-1 is known, the optical flow method can be used for tracking, and feature point extraction and other processes can be performed, that is, there is no need to identify the frame at time t, thereby improving performance. If tracking fails, the recognition and sparse three-dimensional reconstruction processes can be restarted according to the above process.

[0062] Step 103: Send the first information and the audio and video data of the live event to the terminal, so that the terminal generates a three-dimensional virtual live screen of the live event based on the first information and the audio and video data and displays the three-dimensional virtual live screen.

[0063] In the embodiment of the present application, the server obtains images of the live event at different times, locates and tracks the target person and / or the match ball in the image, obtains first information, and sends the first information and audio and video data at different times to the terminal, so that the terminal can use the first information and the audio and video data to obtain the three-dimensional virtual live screen of the live event and display the three-dimensional virtual live screen. Therefore, the terminal can switch the viewing angle at will, thereby improving the user's viewing experience.

[0064] In the above scheme, multiple image acquisition devices are arranged, and technologies such as human body segmentation, human body tracking, and digital human generation are used to achieve a true 360-degree interactive live broadcast of events. Since the above process is real-time, it can meet the requirements of real-time live broadcast, solve the problems of rendering real-time, body positioning, rendering accuracy, etc., and increase the immersive viewing experience of performances and events without the need for VR equipment.

[0065] See also Figure 5 , Figure 5 is a flow chart of a video display method provided in an embodiment of the present application, which is applied to a client, such as Figure 5 As shown, the following steps are included:

[0066] Step 501, obtaining first information sent by a server and audio and video data of a live event at a target time; wherein the first information includes three-dimensional model driving parameters of a target person in the live event and / or three-dimensional position information of a game ball; the server is used to execute the video display method executed by the aforementioned server.

[0067] Specifically, the client includes but is not limited to a terminal.

[0068] Step 502: Obtain a three-dimensional virtual live broadcast picture of the live event according to the first information and the audio and video data.

[0069] In an embodiment of the present application, the three-dimensional virtual live broadcast screen includes a target digital person corresponding to the target person and / or a target three-dimensional game ball corresponding to the game ball.

[0070] Therefore, in this step, the 3D model driving parameters of the target person at the target time can be used to drive a preset 3D human model to obtain the target digital person, and / or the 3D position information of the match ball at the target time can be used to drive a preset 3D spherical model to obtain the target 3D match ball. Afterwards, the audio and video data at the target time are rendered in combination with the target digital person and / or the target 3D match ball to obtain the 3D virtual live broadcast screen.

[0071] In actual applications, after the user opens the live broadcast interface, the virtual environment will be preloaded, such as athlete human models, stadium models, game ball models and other information. After the terminal receives the data stream at the target time, the virtual environment will automatically drive the preset three-dimensional human model to move according to the three-dimensional model driving parameters to obtain the target digital human. In addition, the three-dimensional position information of the game ball at the target time is used to drive the preset three-dimensional spherical model to obtain the target three-dimensional game ball, update the coordinate information of the game ball, and render it. Therefore, terminal users can not only watch the game situation in real time, but also rotate, zoom in and out the virtual live broadcast environment like playing games, enhancing the user's viewing experience. Moreover, compared with the traditional live broadcast data stream, this data stream has a very small data volume and low network speed requirements, and can be used as a supplement to the traditional live broadcast mode.

[0072] Step 503: Display the three-dimensional virtual live broadcast screen.

[0073] While displaying the 3D virtual live screen, an operation interface can also be provided to the user for operation, for example, the size of the screen can be adjusted, the display angle can be rotated, and so on.

[0074] In the embodiment of the present application, the server obtains images of the live event at different times, locates and tracks the target person and / or the match ball in the image, obtains first information, and sends the first information and audio and video data at different times to the terminal, so that the terminal can use the first information and the audio and video data to obtain the three-dimensional virtual live screen of the live event and display the three-dimensional virtual live screen. Therefore, the terminal can switch the viewing angle at will, thereby improving the user's viewing experience.

[0075] See also Figure 6 , Figure 6 is a structural diagram of a video display device provided in an embodiment of the present application, which is applied to a server. Figure 6 As shown, the video display device includes:

[0076] The first acquisition module 601 is used to acquire images of the live event using multiple image acquisition devices set up at the venue of the live event; the second acquisition module 602 is used to locate and track the target person and / or the game ball in the image respectively to obtain first information; wherein the first information includes the three-dimensional model driving parameters of the target person and / or the three-dimensional position information of the game ball; the first sending module 603 is used to send the first information and the audio and video data of the live event to the terminal, so that the terminal generates a three-dimensional virtual live screen of the live event based on the first information and the audio and video data and displays the three-dimensional virtual live screen.

[0077] Optionally, the first information and the audio and video data of the live event are sent to the terminal, so that the terminal generates a three-dimensional virtual live screen of the live event based on the first information and the audio and video data and displays the three-dimensional virtual live screen

[0078] Optionally, the second acquisition module is further used to locate and track target persons in the live event image to obtain three-dimensional model driving parameters of each target person, including:

[0079] A first acquisition submodule is used to determine multiple image sets of a first target person based on images captured by multiple image acquisition devices at a first moment; wherein each image set includes: an RGB image, a depth image, and a corresponding mask image of the first target person, the first moment is any moment during the live event, the first target person is any person among the target persons, and the first target person is any person in the live event;

[0080] A second acquisition submodule is used to obtain multiple skinned multi-person linear model SMPL parameters according to the RGB images and mask images in the multiple image sets;

[0081] A third acquisition submodule, configured to obtain a plurality of scale parameters according to the depth images and the RGB images in the plurality of image sets;

[0082] A fourth acquisition submodule, configured to obtain a plurality of spatial position parameters according to the depth images in the plurality of image sets;

[0083] The fifth acquisition submodule is used to determine the three-dimensional model driving parameters of the first target person at the first moment according to the multiple skinned multi-person linear model SMPL parameters, the multiple scale parameters and the multiple spatial position parameters.

[0084] Optionally, the first acquisition submodule is further used for:

[0085] For each image captured by the image acquisition device at the first moment, human body detection is performed based on the RGB image and the depth image in the image to generate a first bounding box; wherein the depth value of the human body in the first bounding box is less than or equal to a preset value;

[0086] Performing identity recognition on the human body in the first bounding box to determine whether the first target person is included in the first bounding box;

[0087] If the first target person is included in the plurality of first bounding boxes, segmenting the RGB image to obtain a mask image;

[0088] A plurality of image sets of the first target person are formed using a plurality of groups of the RGB images, the depth images, and the mask images.

[0089] Optionally, the fifth acquisition submodule is further used for:

[0090] According to a plurality of skinned multi-person linear model SMPL parameters, a plurality of scale parameters and a plurality of spatial position parameters, a mean value of the SMPL parameters, a mean value of the scale parameters and a mean value of the spatial position parameters are determined;

[0091] Using the SMPL parameter mean, the scale parameter mean, and the spatial position parameter mean to drive a preset three-dimensional human body model corresponding to the first target person, to obtain a candidate three-dimensional human body model;

[0092] Projecting the candidate three-dimensional human body model into a two-dimensional image plane to obtain a two-dimensional outline of the candidate three-dimensional human body model;

[0093] Comparing the two-dimensional outline with the actual outline of the first target person in the RGB image to obtain a comparison result;

[0094] The SMPL parameter mean, the scale parameter mean, and the spatial position parameter mean are adjusted according to the comparison result to obtain the three-dimensional model driving parameters of the first target person at the first moment.

[0095] Optionally, the fifth acquisition submodule is further used for:

[0096] The SMPL parameter mean, the scale parameter mean and the spatial position parameter mean are adjusted by using deep learning back propagation and gradient descent methods to obtain the three-dimensional model driving parameters of the first target person at the first moment.

[0097] The device provided in the embodiment of the present application can execute the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0098] See also Figure 7 , Figure 7 is a structural diagram of a video display device provided in an embodiment of the present application, which is applied to a client. Figure 7 As shown, the video display device includes:

[0099] The first acquisition module 701 is used to acquire the first information sent by the server and the audio and video data of the live event at the target time; wherein the first information includes the three-dimensional model driving parameters of the target person in the live event and / or the three-dimensional position information of the game ball; the server is used to execute the video display method executed by the aforementioned server; the second acquisition module 702 is used to obtain the three-dimensional virtual live broadcast picture of the live event according to the first information and the audio and video data; the first display module 703 is used to display the three-dimensional virtual live broadcast picture.

[0100] The three-dimensional virtual live broadcast screen includes: a target digital person corresponding to the target person and / or a target three-dimensional game ball corresponding to the game ball; the second acquisition module includes:

[0101] A first acquisition submodule is used to drive a preset three-dimensional human body model using the three-dimensional model driving parameters of the target person at the target time to obtain the target digital person, and / or to drive a preset three-dimensional spherical model using the three-dimensional position information of the match ball at the target time to obtain the target three-dimensional match ball;

[0102] The second acquisition submodule is used to render the audio and video data at the target moment in combination with the target digital human and / or the target three-dimensional game ball to obtain the three-dimensional virtual live broadcast picture.

[0103] The device provided in the embodiment of the present application can execute the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0104] It should be noted that the division of units in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0105] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a processor-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0106] See also Figure 8 An embodiment of the present application provides a communication device, including: a memory 801, a processor 802, and a program stored in the memory 801 and executable on the processor 802; the processor 802 is used to read the program in the memory 801 to implement the steps in the video display method as described above.

[0107] The embodiment of the present application also provides a readable storage medium, on which a program is stored, and when the program is executed by the processor, each process of the above-mentioned video display method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. Among them, the readable storage medium can be any available medium or data storage device that can be accessed by the processor, including but not limited to magnetic storage (such as floppy disk, hard disk, magnetic tape, magneto-optical disk (MO), etc.), optical storage (such as CD, DVD, BD, HVD, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)), etc.

[0108] The embodiment of the present application also provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, the various processes of the above-mentioned video display method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, they are not repeated here.

[0109] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0110] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), including a number of instructions for a terminal (which can be a mobile phone, computer, server, air conditioner, or network equipment, etc.) to execute the methods described in each embodiment of the present application.

[0111] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A video display method, characterized in that: Applicable to servers, including: Using a plurality of image acquisition devices disposed at a venue where a live event is broadcast, images of the live event are acquired; Positioning and tracking the target person and / or the match ball in the image respectively to obtain first information; wherein the first information includes the three-dimensional model driving parameters of the target person and / or the three-dimensional position information of the match ball; The first information and the audio and video data of the live event are sent to a terminal, so that the terminal generates a three-dimensional virtual live screen of the live event based on the first information and the audio and video data and displays the three-dimensional virtual live screen.

2. The method according to claim 1, characterized in that The live event image includes: images of the live event collected by the multiple image acquisition devices at multiple identical moments, each of the live event images includes: an RGB image and a depth image, and the three-dimensional model driving parameters of each target person include: skinned multi-person linear model SMPL parameters, scale parameters, and spatial position parameters; Positioning and tracking the target person in the live event image to obtain the three-dimensional model driving parameters of each target person includes: Determine multiple image sets of a first target person based on images captured by multiple image acquisition devices at a first moment; wherein each image set includes: an RGB image, a depth image, and a corresponding mask image of the first target person, the first moment is any moment during the live event, the first target person is any person among the target persons, and the first target person is any person in the live event; According to the RGB images and mask images in the plurality of image sets, a plurality of skinned multi-person linear model SMPL parameters are obtained; Obtaining multiple scale parameters according to the depth images and the RGB images in the multiple image sets; Obtaining a plurality of spatial position parameters according to the depth images in the plurality of image sets; Determine the three-dimensional model driving parameters of the first target person at the first moment according to the multiple skinned multi-person linear model SMPL parameters, the multiple scale parameters, and the multiple spatial position parameters.

3. The method according to claim 2, characterized in that The step of determining a plurality of image sets of a first target person based on images captured by a plurality of image acquisition devices at a first moment includes: For each image captured by the image acquisition device at the first moment, human body detection is performed based on the RGB image and the depth image in the image to generate a first bounding box; wherein the depth value of the human body in the first bounding box is less than or equal to a preset value; Performing identity recognition on the human body in the first bounding box to determine whether the first target person is included in the first bounding box; If the first target person is included in the plurality of first bounding boxes, segmenting the RGB image to obtain a mask image; A plurality of image sets of the first target person are formed using a plurality of groups of the RGB images, the depth images, and the mask images.

4. The method according to claim 2, characterized in that: The determining, according to the plurality of skinned multi-person linear model SMPL parameters, the plurality of scale parameters, and the plurality of spatial position parameters, the three-dimensional model driving parameters of the first target person at the first moment includes: According to a plurality of skinned multi-person linear model SMPL parameters, a plurality of scale parameters and a plurality of spatial position parameters, a mean value of the SMPL parameters, a mean value of the scale parameters and a mean value of the spatial position parameters are determined; Using the SMPL parameter mean, the scale parameter mean, and the spatial position parameter mean to drive a preset three-dimensional human body model corresponding to the first target person, to obtain a candidate three-dimensional human body model; Projecting the candidate three-dimensional human body model into a two-dimensional image plane to obtain a two-dimensional outline of the candidate three-dimensional human body model; Comparing the two-dimensional outline with the actual outline of the first target person in the RGB image to obtain a comparison result; The SMPL parameter mean, the scale parameter mean, and the spatial position parameter mean are adjusted according to the comparison result to obtain the three-dimensional model driving parameters of the first target person at the first moment.

5. The method according to claim 4, characterized in that The step of adjusting the SMPL parameter mean, the scale parameter mean, and the spatial position parameter mean according to the comparison result to obtain the three-dimensional model driving parameters of the first target person at the first moment includes: The SMPL parameter mean, the scale parameter mean and the spatial position parameter mean are adjusted by using deep learning back propagation and gradient descent methods to obtain the three-dimensional model driving parameters of the first target person at the first moment.

6. A video display method, characterized in that: Applied to the client, including: Acquire first information sent by a server and audio and video data of a live event at a target time; wherein the first information includes three-dimensional model driving parameters of a target person in the live event and / or three-dimensional position information of a match ball; the server is used to execute the video display method described in any one of claims 1 to 5; Obtaining a three-dimensional virtual live screen of the live event according to the first information and the audio and video data; The three-dimensional virtual live broadcast screen is displayed.

7. The method according to claim 6, characterized in that The three-dimensional virtual live broadcast screen includes: a target digital person corresponding to the target person and / or a target three-dimensional game ball corresponding to the game ball; The step of obtaining a three-dimensional virtual live broadcast picture of the live event according to the first information and the audio and video data includes: Using the three-dimensional model driving parameters of the target person at the target time to drive a preset three-dimensional human body model to obtain the target digital person, and / or using the three-dimensional position information of the match ball at the target time to drive a preset three-dimensional sphere model to obtain the target three-dimensional match ball; The audio and video data at the target moment are rendered in combination with the target digital human and / or the target three-dimensional game ball to obtain the three-dimensional virtual live broadcast picture.

8. An electronic device comprising: A memory, a processor, and a program stored in the memory and executable on the processor; wherein the processor is used to read the program in the memory to implement the steps in the video display method as described in any one of claims 1 to 7.

9. A computer-readable storage medium for storing a program, characterized in that: When the program is executed by a processor, the steps of the video display method according to any one of claims 1 to 7 are implemented.

10. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps in the video display method according to any one of claims 1 to 7.

Images