A video playback method, system, device and medium based on 3D dynamic rendering

Abstract

The present application provides a video playback method, system, device and medium based on 3D dynamic rendering, wherein the first cloud server sends a video rendering file to the user terminal; the user terminal renders part of the video according to the video rendering file to obtain a first video clip for playback; when the first video clip is played, the user terminal collects the position where the user's line of sight falls in several pictures of the first video clip and counts the duration of the stay; the second cloud server determines the user's focus area and target based on the stay position and duration, and updates the 3D rendering effect parameters based on the focus area and target; after the first cloud server renders the screen according to the updated rendering parameters to obtain a second video clip, the user terminal plays the second video clip to achieve the playback of the target video. The above method is used to reduce the computing power and computing resources required for rendering and playing the video, and ensure the normal rendering and playback of the video.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a video playback method, system, device and medium based on 3D dynamic rendering. Background Art

[0002] With the continuous improvement of computer technology, 3D movies, 3D animations, 3D games, mobile stereo navigation, AR (Augmented Reality) and other 3D interactive information are becoming increasingly popular, and 3D content continues to be enriched, constantly bringing new visual impact and immersive experience to users. In the prior art, when rendering these videos, all images are usually rendered at the same resolution, and high-resolution real-time video encoding is performed to obtain 3D videos.

[0003] However, the study found that 3D movies or some other 3D videos may contain complex stereoscopic scenes and a large amount of picture content. When rendering these 3D videos using the above rendering method, a large amount of computing resources and storage space are required, which in turn causes excessive load on the computer device. For some devices with poor processing capabilities, it is impossible to render the video screen and play the rendered video normally. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a video playback method, system, device and medium based on 3D dynamic rendering, so as to reduce the computing power and computing resources required for rendering and playing the video, and ensure the normal rendering and playback of the video.

[0005] In a first aspect, an embodiment of the present application provides a video playback method based on 3D dynamic rendering, which is applied to a video playback system based on 3D dynamic rendering, wherein the system includes a user terminal, a first cloud server and a second cloud server, and the method includes:

[0006] The first cloud server responds to the video playback instruction of the user terminal and sends a video rendering file to the user terminal, wherein the video rendering file includes at least part of the video data of the target video indicated by the video playback instruction and initial 3D rendering effect parameters, and the initial 3D rendering effect parameters include a rendering range and a rendering accuracy;

[0007] The user terminal performs picture rendering based on at least part of the video data according to the initial 3D rendering effect parameters to obtain a first video segment of the target video, and plays the first video segment to the user through a video playback sub-interface in the user interface;

[0008] When the first video segment is played, the user terminal collects the user's sight point on the user interface in real time, and determines the stop position of the user's sight point in several first segment images of the first video segment based on the position of the interface sight point and the position and area of ​​the video playback sub-interface in the user interface;

[0009] The user terminal counts the duration of the sight line of the picture at each stop position, and sends the stop position and the stop duration of the sight line of the picture in each first segment picture to the second cloud server;

[0010] The second cloud server determines the user's attention area and attention target in the first video segment based on the stay position and the stay duration of the screen sight point in each first segment screen, and determines whether the user's attention area belongs to the attention target;

[0011] If the user's attention area belongs to the attention target, the second cloud server updates the rendering range in the initial 3D rendering effect parameters by using the area where the attention target is located to obtain the target 3D rendering effect parameters; if the attention area does not belong to the attention target, the second cloud server updates the rendering range in the initial 3D rendering effect parameters by using the attention area and the area where the attention target is located to obtain the target 3D rendering effect parameters;

[0012] Alternatively, if the user's focus area belongs to the focus target, the second cloud server lowers the rendering accuracy of other non-target areas other than the focus target to obtain the target 3D rendering effect parameters; if the focus area does not belong to the focus target, the second cloud server lowers the rendering accuracy of non-focus areas other than the focus area and the non-target area to obtain the target 3D rendering effect parameters;

[0013] The first cloud server performs screen rendering based on the target video data according to the target 3D rendering effect parameter to obtain a second video segment of the target video except the first video segment, and sends the second video segment to the user terminal, wherein the target video data is the video data of the target video except the at least part of the video data;

[0014] After the first video segment is played, the user terminal continues to play the second video segment to the user, thereby achieving the playback of the target video.

[0015] Optionally, determining the user's focus area in the first video segment based on the stay position and stay duration of the sight point in each first segment screen includes:

[0016] For each first segment picture, perform human body detection on the first segment picture using a human body detection model, and divide the first segment picture into human body areas and non-human body areas based on the human body detection result;

[0017] Performing face detection on the human body region in the first image segment using a face detection model, and dividing the human body region into a face region and a non-face region based on the face detection result;

[0018] Counting the dwell time of each segmented area where the sight point of the screen stays, and determining whether the dwell time corresponding to each segmented area exceeds a preset area time, wherein the segmented areas include the face area, the non-face area, and the non-human body area;

[0019] Determine the segmented area whose corresponding area stay time exceeds the preset area time as the focus sub-area in the first segment picture;

[0020] The user's focus area in the first video segment is determined based on the focus sub-area in each first segment screen.

[0021] Optionally, determining the user's attention target in the first video segment based on the stay position and stay duration of the sight point in each first segment screen includes:

[0022] For each first segment picture, using a human body detection model to perform human body recognition on the first segment picture, and dividing the first segment picture into human body areas and non-human body areas based on the human body recognition results;

[0023] Performing face recognition on the human body regions in the first image segment using a face recognition model, and determining the person indicated by each human body region based on the face recognition result;

[0024] Counting the duration of a person's stay in each human body region at the position where the sight line of the screen falls, and determining whether the person corresponding to each human body region stays for more than a preset duration;

[0025] Determine the person indicated by the human body region whose corresponding person stays for a time period exceeding the preset person stay time as a focus sub-target in the first segment picture;

[0026] The focus target of the user in the first video segment is determined based on the focus sub-targets in each first segment screen.

[0027] Optionally, the determining, based on the position of the sight point of the interface and according to the position and area of ​​the video playback sub-interface in the user interface, the stopping position of the sight point of the user in the first segment screens of the first video segment includes:

[0028] Determining an offset of the video playback sub-interface relative to the user interface according to a position of the video playback sub-interface in the user interface;

[0029] Determining a scaling ratio of the video playback sub-interface relative to the user interface according to an area of ​​the video playback sub-interface in the user interface;

[0030] For each first segment picture, based on the position of the interface sight point when playing the first segment picture, the stop position of the screen sight point in the first segment picture is determined according to the offset and the scaling ratio.

[0031] In a second aspect, an embodiment of the present application provides a video playback system based on 3D dynamic rendering, the system comprising a user terminal, a first cloud server and a second cloud server;

[0032] The first cloud server is configured to respond to a video playback instruction of the user terminal and send a video rendering file to the user terminal, wherein the video rendering file includes at least part of the video data of the target video indicated by the video playback instruction and initial 3D rendering effect parameters, and the initial 3D rendering effect parameters include a rendering range and a rendering accuracy;

[0033] The user terminal is configured to perform picture rendering based on at least part of the video data according to the initial 3D rendering effect parameters to obtain a first video segment of the target video, and play the first video segment to the user through a video playback sub-interface in the user interface;

[0034] The user terminal is used to collect the user's sight point on the user interface in real time when the first video segment is played, and determine the stop position of the user's sight point in the first segment screens of the first video segment based on the position of the interface sight point and the position and area of ​​the video playback sub-interface in the user interface;

[0035] The user terminal is used to count the duration of the sight line of the picture at each stop position, and send the stop position and the stop duration of the sight line of the picture in each first segment picture to the second cloud server;

[0036] The second cloud server is used to determine the user's attention area and attention target in the first video segment based on the stay position and the stay duration of the screen sight point in each first segment screen, and determine whether the user's attention area belongs to the attention target;

[0037] The second cloud server is configured to update the rendering range in the initial 3D rendering effect parameters using the area where the target of interest is located to obtain the target 3D rendering effect parameters if the user's attention area belongs to the target of interest; and to update the rendering range in the initial 3D rendering effect parameters using the attention area and the area where the target of interest is located to obtain the target 3D rendering effect parameters if the attention area does not belong to the target of interest;

[0038] Alternatively, the second cloud server is configured to, if the user's focus area belongs to the focus target, lower the rendering accuracy of other non-target areas other than the area where the focus target is located to obtain the target 3D rendering effect parameters; if the focus area does not belong to the focus target, lower the rendering accuracy of non-focus areas other than the focus area and the non-target area to obtain the target 3D rendering effect parameters;

[0039] The first cloud server is configured to perform picture rendering based on the target video data according to the target 3D rendering effect parameter to obtain a second video segment of the target video except the first video segment, and send the second video segment to the user terminal, wherein the target video data is the video data of the target video except the at least part of the video data;

[0040] The user terminal is used to continue playing the second video segment to the user after the first video segment is played, so as to achieve the playback of the target video.

[0041] Optionally, determining the user's focus area in the first video segment based on the stay position and stay duration of the sight point in each first segment screen includes:

[0042] For each first segment picture, perform human body detection on the first segment picture using a human body detection model, and divide the first segment picture into human body areas and non-human body areas based on the human body detection result;

[0043] Performing face detection on the human body region in the first image segment using a face detection model, and dividing the human body region into a face region and a non-face region based on the face detection result;

[0044] Counting the dwell time of each segmented area where the sight point of the screen stays, and determining whether the dwell time corresponding to each segmented area exceeds a preset area time, wherein the segmented areas include the face area, the non-face area, and the non-human body area;

[0045] Determine the segmented area whose corresponding area stay time exceeds the preset area time as the focus sub-area in the first segment picture;

[0046] The user's focus area in the first video segment is determined based on the focus sub-area in each first segment screen.

[0047] Optionally, determining the user's attention target in the first video segment based on the stay position and stay duration of the sight point in each first segment screen includes:

[0048] For each first segment picture, using a human body detection model to perform human body recognition on the first segment picture, and dividing the first segment picture into human body areas and non-human body areas based on the human body recognition results;

[0049] Performing face recognition on the human body regions in the first image segment using a face recognition model, and determining the person indicated by each human body region based on the face recognition result;

[0050] Counting the duration of a person's stay in each human body region at the position where the sight line of the screen falls, and determining whether the person corresponding to each human body region stays for more than a preset duration;

[0051] Determine the person indicated by the human body region whose corresponding person stays for a time period exceeding the preset person stay time as a focus sub-target in the first segment picture;

[0052] The focus target of the user in the first video segment is determined based on the focus sub-targets in each first segment screen.

[0053] Optionally, the determining, based on the position of the sight point of the interface and according to the position and area of ​​the video playback sub-interface in the user interface, the stopping position of the sight point of the user in the first segment screens of the first video segment includes:

[0054] Determining an offset of the video playback sub-interface relative to the user interface according to a position of the video playback sub-interface in the user interface;

[0055] Determining a scaling ratio of the video playback sub-interface relative to the user interface according to an area of ​​the video playback sub-interface in the user interface;

[0056] For each first segment picture, based on the position of the interface sight point when playing the first segment picture, the stop position of the screen sight point in the first segment picture is determined according to the offset and the scaling ratio.

[0057] In a third aspect, an embodiment of the present application provides a computer device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor communicates with the memory through the bus, and when the machine-readable instructions are executed by the processor, the steps of the video playback method based on 3D dynamic rendering described in any optional implementation manner of the second aspect are performed.

[0058] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the video playback method based on 3D dynamic rendering described in any optional implementation manner of the second aspect above are executed.

[0059] The technical solution provided by this application includes but is not limited to the following beneficial effects:

[0060] This application divides the screen rendering of the target video into two stages through a video playback system based on 3D dynamic rendering: in the first stage, the user terminal performs screen rendering of a small part of the video based on the initial 3D rendering effect parameters to obtain the first video segment to be played; in the second stage, the second cloud server determines the user's focus area and focus target based on the user's sight point when watching the first video segment, and dynamically updates and adjusts the initial 3D rendering effect parameters based on the user's focus area and focus target, and then the first cloud server performs screen rendering on most of the remaining video data according to the adjusted 3D rendering effect parameters to obtain the second video segment, and then sends it to the user terminal for playback. Through the above two stages, the rendering work of the two video segments of the target video is respectively allocated to the user terminal and the cloud server, which can realize the reasonable sharing of computing resources and reduce the burden of computing power of a single device. In addition, based on the user's focus on the screen when watching the first video segment, the rendering effect parameters of the second video segment are dynamically adjusted and updated, the rendering range is narrowed in a targeted manner, and the rendering accuracy is adaptively reduced, which can reduce the amount of calculation in the video rendering process, reduce the occupation of computing resources, and ensure the normal progress of video rendering. Finally, the video clips obtained by local rendering and cloud rendering with different rendering effect parameters are played continuously, so that the user terminal can play them continuously with the playback effect adapted to the user's attention, thereby improving the user experience when watching videos.

[0061] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0063] Figure 1 A flowchart of a video playback method based on 3D dynamic rendering provided by the first embodiment of the present invention is shown;

[0064] Figure 2 A flow chart of a method for determining a region of interest provided by Embodiment 1 of the present invention is shown;

[0065] Figure 3 A flow chart of a method for determining a target of interest provided by the first embodiment of the present invention is shown;

[0066] Figure 4 A flow chart of a method for determining a stop position provided by Embodiment 1 of the present invention is shown;

[0067] Figure 5 A schematic diagram of the structure of a video playback system based on 3D dynamic rendering provided by the second embodiment of the present invention is shown;

[0068] Figure 6 A schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention is shown. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention.

[0070] Embodiment 1

[0071] To facilitate understanding of this application, Figure 1 The flowchart of a video playback method based on 3D dynamic rendering provided by the first embodiment of the present invention is described in detail to explain the first embodiment of the present application.

[0072] See also Figure 1 As shown, Figure 1 A flowchart of a video playback method based on 3D dynamic rendering provided by Embodiment 1 of the present invention is shown, wherein the method is applied to a video playback system based on 3D dynamic rendering, the system includes a user terminal, a first cloud server and a second cloud server, and the method includes steps S101 to S109:

[0073] S101: The first cloud server responds to a video playback instruction of the user terminal and sends a video rendering file to the user terminal, wherein the video rendering file includes at least part of the video data of the target video indicated by the video playback instruction and initial 3D rendering effect parameters, and the initial 3D rendering effect parameters include a rendering range and a rendering accuracy.

[0074] Specifically, at least part of the video data of the target video is part of the video data in the target video data. For example, if the target video is a movie, at least part of the video data may be the video data of the movie title, or the video data of the first minute. The target video may also be a game, a short video, a TV series, a movie, etc. The initial 3D rendering effect parameters include a rendering range and a rendering accuracy, and may also include a rendering duration. The rendering range of the initial 3D rendering effect parameters is global rendering, and the rendering accuracy is the highest accuracy. Typically, the amount of data of at least part of the video data is less than the amount of data of the remaining video data.

[0075] After the user terminal sends a video play instruction to the first cloud server, the first cloud server responds to the instruction and sends a video rendering file to the user terminal, so that the user terminal can perform local video rendering according to the video data and rendering effect parameters.

[0076] S102: The user terminal performs screen rendering based on at least a portion of the video data according to the initial 3D rendering effect parameters to obtain a first video segment of the target video, and plays the first video segment to the user through a video playback sub-interface in a user interface.

[0077] Specifically, the user terminal performs local rendering to obtain a first video segment, which is a part of the target video. The user terminal does not need to render the entire video, which can reduce the consumption of local computing resources of the user terminal.

[0078] S103: When the first video segment is playing, the user terminal collects the user's line of sight on the user interface in real time, and based on the position of the line of sight, determines the stop position of the user's line of sight in several first segment images of the first video segment according to the position and area of ​​the video playback sub-interface in the user interface.

[0079] Specifically, since different users may be interested in different things, when the user terminal plays the first video clip to the user through the video playback sub-interface, the areas or things that each user is looking at and paying attention to will be different. Considering that global rendering usually requires a large amount of calculation, in order to reduce the rendering calculation amount of the target video and save computing resources, this application associates the user's focus area and focus target with the rendering strategy, and adjusts the subsequent rendering range and rendering accuracy based on the user's focus area and focus target.

[0080] In this step, the user terminal collects the user's line of sight in the user interface in real time. Since the user may be playing the video in full screen through the user interface (at this time the video playback sub-interface and the user interface completely overlap), or may be playing the video in a small screen in a video playback sub-interface that does not fill the entire user interface, it is necessary to convert the user's interface stay position in the user interface to the video playback sub-interface based on the position and area of ​​the video playback sub-interface in the user interface, and obtain the stay position of the line of sight in several first segment pictures of the first video segment.

[0081] S104: The user terminal counts the duration of the sight line of the picture at each stop position, and sends the stop position and the stop duration of the sight line of the picture in each first segment picture to the second cloud server.

[0082] Specifically, the user terminal counts the stay position and stay duration of the screen sight points in each first screen segment, and sends the statistics to the second cloud server for data analysis.

[0083] S105: The second cloud server determines the user's focus area and focus target in the first video segment based on the stay position and stay duration of the screen sight point in each first segment screen, and determines whether the user's focus area belongs to his focus target.

[0084] Specifically, the present application distinguishes the functions of the first cloud server and the second cloud server. The first cloud server is a rendering server, which is mainly used for image rendering; the second cloud server is a data processing server, which is mainly used for other tasks besides image rendering. The focus area includes the human body area and the non-human body area, and the human body area includes the face area and the non-face area. The focus target is a specific person or thing.

[0085] For each first segment, the second server determines the user's focus area and focus target in the first video segment based on the stay position and stay duration of the sight point in the first segment. The longer the user's sight point stays on a certain area or target, the more attention the user pays to the area or target.

[0086] For example, the first segment includes the face regions of Person A, Person B, and Person C. If the second cloud server determines that the user's focus region is Person A's face region and the focus target is Person A, then by judging whether Person A's face region belongs to Person A, it can be concluded that the user's focus region belongs to the focus target. If the second cloud server determines that the user's focus region is Person A's face region and the focus target is Person B, then by judging whether Person A's face region belongs to Person B, it can be concluded that the user's focus region does not belong to the focus target.

[0087] S106: If the user's focus area belongs to the target of interest, the second cloud server uses the area where the target of interest is located to update the rendering range in the initial 3D rendering effect parameters to obtain the target 3D rendering effect parameters; if the focus area does not belong to the target of interest, the second cloud server uses the focus area and the area where the target of interest is located to update the rendering range in the initial 3D rendering effect parameters to obtain the target 3D rendering effect parameters.

[0088] Specifically, there are two ways to adjust the rendering effect parameters. The first way is to adjust only the rendering range, and the second way is to adjust both the rendering range and the rendering accuracy. The following is the first way to adjust the rendering effect parameters:

[0089] If the user's focus area belongs to the target of interest, it means that the user is very concerned about the target of interest, and the area where the target of interest is located is more concerned by the user than other areas. Therefore, the rendering range in the initial 3D rendering effect parameters is updated to the area where the target of interest is located to obtain the target 3D rendering effect parameters. For example, if the user's focus area is the face area of ​​Person A, and the determined target of interest is Person A, the rendering range in the initial 3D rendering effect parameters is updated to the area where Person A is located (including Person A's body and face), that is, the rendering range is adjusted from global rendering to local rendering in a targeted manner, and only the area where Person A is located is rendered.

[0090] If the user's focus area does not belong to the focus target, it means that the user is paying attention to the focus area in addition to the focus target, so the rendering range in the initial 3D rendering effect parameters is updated to the focus area and the area where the focus target is located to obtain the target 3D rendering effect parameters. For example, if the user's focus area is the face area of ​​Person A, and the determined focus target is Person B, the rendering range in the initial 3D rendering effect parameters is updated to the face area of ​​Person A and the area where Person B is located (including Person B's body and face), that is, the rendering range is adjusted from global rendering to local rendering in a targeted manner, and only the face area of ​​Person A and the area where Person B is located are rendered.

[0091] S107: Alternatively, if the user's focus area belongs to the target of interest, the second cloud server lowers the rendering accuracy of other non-target areas outside the area where the target of interest is located to obtain the target 3D rendering effect parameters; if the focus area does not belong to the target of interest, the second cloud server lowers the rendering accuracy of non-focus areas and the non-target areas outside the focus area to obtain the target 3D rendering effect parameters.

[0092] Specifically, the second method of adjusting rendering effect parameters is as follows:

[0093] If the user's focus area belongs to the target of interest, the rendering accuracy of other non-target areas other than the area where the target of interest is located is lowered, the original rendering accuracy of the area where the target of interest is located is maintained, and the target 3D rendering effect parameters are obtained. For example, if the user's focus area is the face area of ​​Person A, and the determined target of interest is Person A, the rendering accuracy of other areas other than the area where Person A is located (including Person A's body and face) in the initial 3D rendering effect parameters is lowered, and only the area where the target of interest to the user is located is retained for high-precision rendering, and the rendering accuracy of the remaining areas is reduced.

[0094] If the user's focus area does not belong to the focus target, the rendering accuracy of the non-focus area and the non-target area outside the focus area is lowered, and only the original rendering accuracy of the focus area and the area where the focus target is located is maintained to obtain the target 3D rendering effect parameters. For example, if the user's focus area is the face area of ​​Person A, and the determined focus target is Person B, the rendering accuracy of other target areas other than the area where Person B is located (including Person A's body and face) and other focus areas other than Person A's face area in the initial 3D rendering effect parameters is lowered, and only the focus area of ​​interest to the user and the area where the focus target is located are retained for high-precision rendering, and the rendering accuracy of the remaining areas is reduced.

[0095] S108: The first cloud server performs screen rendering based on the target video data according to the target 3D rendering effect parameters to obtain a second video segment of the target video excluding the first video segment, and sends the second video segment to the user terminal, wherein the target video data is the video data of the target video excluding at least part of the video data.

[0096] Specifically, the target video data is the remaining video data of the target video except for at least part of the video data rendered by the user terminal. At least part of the video data rendered by the user terminal corresponds to the first video segment, and this part of the target video data corresponds to the second video segment. Considering that the second video segment is longer and has more content, the amount of calculation required for its image rendering and the computing resources occupied are also greater, so the first cloud server executes the rendering task of the second video segment, performs cloud rendering, and renders the image based on the target video data according to the target 3D rendering effect parameters to obtain the second video segment, thereby reducing the local computing resource occupation of the user terminal. Usually, the duration of the first video segment is less than the duration of the second video segment.

[0097] S109: After the first video segment is played, the user terminal continues to play the second video segment to the user, thereby playing the target video.

[0098] Specifically, after the first video segment is played, the user terminal continues to play the second video segment to the user to achieve continuous playback of the target video.

[0099] In one possible embodiment, see Figure 2 As shown, Figure 2 A flowchart of a method for determining an area of ​​interest provided by a first embodiment of the present invention is shown, wherein the area of ​​interest of the user in the first video segment is determined based on the stay position and stay duration of the sight point in each first segment, including steps S201 to S205:

[0100] S201: For each first segment picture, perform human body detection on the first segment picture using a human body detection model, and divide the first segment picture into a human body area and a non-human body area based on the human body detection result.

[0101] S202: Performing face detection on the human body region in the first image segment using a face detection model, and dividing the human body region into a face region and a non-face region based on the face detection result.

[0102] S203: Counting the length of time that the position where the sight line of sight falls on the screen stays in each segmented area, and determining whether the length of time that the segmented area corresponds to each segmented area exceeds a preset length of time, wherein the segmented areas include the face area, the non-face area, and the non-human body area.

[0103] S204: Determine the segmented area whose corresponding area stay time exceeds the preset area stay time as the focus sub-area in the first segment picture.

[0104] Specifically, the segmented area with the longest corresponding area stay time is determined as the focus sub-area in the first segment picture.

[0105] S205: Determine the user's focus area in the first video segment based on the focus sub-areas in each first segment screen.

[0106] Specifically, the attention sub-region that appears most times in each first segment screen is used as the attention sub-region of the user in the first video segment.

[0107] In one possible embodiment, see Figure 3 As shown, Figure 3 A flowchart of a method for determining a focus target provided by the first embodiment of the present invention is shown, wherein the focus target of the user in the first video segment is determined based on the stay position and stay duration of the sight point in each first segment screen, including steps S301 to S305:

[0108] S301: For each first segment picture, perform human body recognition on the first segment picture by using a human body detection model, and divide the first segment picture into a human body area and a non-human body area based on the human body recognition result.

[0109] S302: Performing face recognition on the human body regions in the first image segment using a face recognition model, and determining the person indicated by each human body region based on the face recognition result.

[0110] S303: Counting the duration of a person's stay in each human body region where the sight line of the screen falls, and determining whether the person's stay in each human body region exceeds a preset duration.

[0111] S304: Determine the person indicated by the human body region whose corresponding person stays for a time period exceeding the preset person stay time as the focus sub-target in the first segment image.

[0112] Specifically, the person indicated by the human body region where the corresponding person stays for the longest time is determined as the focus sub-target in the first segment image.

[0113] S305: Determine the focus target of the user in the first video segment based on the focus sub-targets in each first segment screen.

[0114] Specifically, the attention sub-target that appears most times in each first segment screen is used as the attention target of the user in the first video segment.

[0115] In one possible embodiment, see Figure 4 As shown, Figure 4 A flow chart of a method for determining a dwell position provided in Embodiment 1 of the present invention is shown, wherein the method determines the dwell position of the user's visual line of sight in a plurality of first segment images of the first video segment based on the position of the visual line of sight of the interface and the position and area of ​​the video playback sub-interface in the user interface, including steps S401 to S403:

[0116] S401: Determine an offset of the video playback sub-interface relative to the user interface according to a position of the video playback sub-interface in the user interface.

[0117] Specifically, the first coordinate of the center point of the video playback sub-interface and the second coordinate of the center point of the user interface are obtained, and the target offset direction and target offset amount of the first coordinate compared to the second coordinate are calculated.

[0118] S402: Determine a scaling ratio of the video playback sub-interface relative to the user interface according to the area of ​​the video playback sub-interface in the user interface.

[0119] Specifically, a first area of ​​the video playback sub-interface and a second area of ​​the user interface are calculated, and the first area is divided by the second area to obtain a target zoom ratio.

[0120] S403: For each first segment picture, based on the position of the interface sight point when playing the first segment picture, determine the stop position of the screen sight point in the first segment picture according to the offset and the scaling ratio.

[0121] Specifically, the coordinates of the interface sight point are offset in the target offset direction to meet the target offset amount, and then the coordinate values ​​of the interface sight point after the offset processing are scaled by the target scaling ratio to obtain new coordinate values, and the position indicated by the new coordinate value is used as the stop position of the screen sight point.

[0122] Embodiment 2

[0123] See also Figure 5 As shown, Figure 5 A schematic diagram of the structure of a video playback system based on 3D dynamic rendering provided by Embodiment 2 of the present invention is shown, wherein the system includes a user terminal 501, a first cloud server 502 and a second cloud server 503;

[0124] The first cloud server is configured to respond to a video playback instruction of the user terminal and send a video rendering file to the user terminal, wherein the video rendering file includes at least part of the video data of the target video indicated by the video playback instruction and initial 3D rendering effect parameters, and the initial 3D rendering effect parameters include a rendering range and a rendering accuracy;

[0125] The user terminal is configured to perform picture rendering based on at least part of the video data according to the initial 3D rendering effect parameters to obtain a first video segment of the target video, and play the first video segment to the user through a video playback sub-interface in the user interface;

[0126] The user terminal is used to collect the user's sight point on the user interface in real time when the first video segment is played, and determine the stop position of the user's sight point in the first segment screens of the first video segment based on the position of the interface sight point and the position and area of ​​the video playback sub-interface in the user interface;

[0127] The user terminal is used to count the duration of the sight line of the picture at each stop position, and send the stop position and the stop duration of the sight line of the picture in each first segment picture to the second cloud server;

[0128] The second cloud server is used to determine the user's attention area and attention target in the first video segment based on the stay position and the stay duration of the screen sight point in each first segment screen, and determine whether the user's attention area belongs to the attention target;

[0129] The second cloud server is configured to update the rendering range in the initial 3D rendering effect parameters using the area where the target of interest is located to obtain the target 3D rendering effect parameters if the user's attention area belongs to the target of interest; and to update the rendering range in the initial 3D rendering effect parameters using the attention area and the area where the target of interest is located to obtain the target 3D rendering effect parameters if the attention area does not belong to the target of interest;

[0130] Alternatively, the second cloud server is configured to, if the user's focus area belongs to the focus target, lower the rendering accuracy of other non-target areas other than the area where the focus target is located to obtain the target 3D rendering effect parameters; if the focus area does not belong to the focus target, lower the rendering accuracy of non-focus areas other than the focus area and the non-target area to obtain the target 3D rendering effect parameters;

[0131] The first cloud server is configured to perform picture rendering based on the target video data according to the target 3D rendering effect parameter to obtain a second video segment of the target video except the first video segment, and send the second video segment to the user terminal, wherein the target video data is the video data of the target video except the at least part of the video data;

[0132] The user terminal is used to continue playing the second video segment to the user after the first video segment is played, so as to achieve the playback of the target video.

[0133] In a feasible implementation manner, determining the user's focus area in the first video segment based on the stay position and stay duration of the sight point in each first segment screen includes:

[0134] For each first segment picture, perform human body detection on the first segment picture using a human body detection model, and divide the first segment picture into human body areas and non-human body areas based on the human body detection result;

[0135] Performing face detection on the human body region in the first image segment using a face detection model, and dividing the human body region into a face region and a non-face region based on the face detection result;

[0136] Counting the dwell time of each segmented area where the sight point of the screen stays, and determining whether the dwell time corresponding to each segmented area exceeds a preset area time, wherein the segmented areas include the face area, the non-face area, and the non-human body area;

[0137] Determine the segmented area whose corresponding area stay time exceeds the preset area time as the focus sub-area in the first segment picture;

[0138] The user's focus area in the first video segment is determined based on the focus sub-area in each first segment screen.

[0139] In a feasible implementation manner, determining the user's focus target in the first video segment based on the stay position and stay duration of the sight point in each first segment screen includes:

[0140] For each first segment picture, using a human body detection model to perform human body recognition on the first segment picture, and dividing the first segment picture into human body areas and non-human body areas based on the human body recognition results;

[0141] Performing face recognition on the human body regions in the first image segment using a face recognition model, and determining the person indicated by each human body region based on the face recognition result;

[0142] Counting the duration of a person's stay in each human body region at the position where the sight line of the screen falls, and determining whether the person corresponding to each human body region stays for more than a preset duration;

[0143] Determine the person indicated by the human body region whose corresponding person stays for a time period exceeding the preset person stay time as a focus sub-target in the first segment picture;

[0144] The focus target of the user in the first video segment is determined based on the focus sub-targets in each first segment screen.

[0145] In a feasible implementation manner, the method of determining the position of the user's sight line landing point in a plurality of first segment images of the first video segment based on the position of the interface sight line landing point and according to the position and area of ​​the video playback sub-interface in the user interface includes:

[0146] Determining an offset of the video playback sub-interface relative to the user interface according to a position of the video playback sub-interface in the user interface;

[0147] Determining a scaling ratio of the video playback sub-interface relative to the user interface according to an area of ​​the video playback sub-interface in the user interface;

[0148] For each first segment picture, based on the position of the interface sight point when playing the first segment picture, the stop position of the screen sight point in the first segment picture is determined according to the offset and the scaling ratio.

[0149] Embodiment 3

[0150] Based on the same application concept, see Figure 6 As shown, Figure 6 FIG. 4 shows a schematic diagram of the structure of a computer device provided by Embodiment 3 of the present invention, wherein: Figure 6 As shown, a computer device 600 provided in Embodiment 3 of the present application includes:

[0151] A processor 601, a memory 602 and a bus 603, wherein the memory 602 stores machine-readable instructions executable by the processor 601. When the computer device 600 is running, the processor 601 communicates with the memory 602 via the bus 603. When the processor 601 is running, the machine-readable instructions execute the steps of the video playback method based on 3D dynamic rendering shown in the above-mentioned embodiment 1.

[0152] Embodiment 4

[0153] Based on the same application concept, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the video playback method based on 3D dynamic rendering described in any one of the above embodiments are executed.

[0154] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0155] The computer program product for performing video playback based on 3D dynamic rendering provided in an embodiment of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the method described in the previous method embodiment. The specific implementation can be found in the method embodiment, which will not be repeated here.

[0156] The video playback system based on 3D dynamic rendering provided in the embodiment of the present invention can be specific hardware on the device or software or firmware installed on the device. The system provided in the embodiment of the present invention has the same implementation principle and technical effects as the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the system embodiment, reference can be made to the corresponding contents in the aforementioned method embodiment. Technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can all refer to the corresponding processes in the aforementioned method embodiment, and will not be repeated here.

[0157] In the embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The system embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some communication interface, device or unit, which can be electrical, mechanical or other forms.

[0158] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0159] In addition, each functional unit in the embodiment provided by the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0160] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a service device, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0161] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0162] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can still modify the technical solutions described in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or perform equivalent replacements on some of the technical features thereof; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. They should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A video playback method based on 3D dynamic rendering, characterized in that: Applied to a video playback system based on 3D dynamic rendering, the system includes a user terminal, a first cloud server and a second cloud server, and the method includes: The first cloud server responds to the video playback instruction of the user terminal and sends a video rendering file to the user terminal, wherein the video rendering file includes at least part of the video data of the target video indicated by the video playback instruction and initial 3D rendering effect parameters, and the initial 3D rendering effect parameters include a rendering range and a rendering accuracy; The user terminal performs picture rendering based on at least part of the video data according to the initial 3D rendering effect parameters to obtain a first video segment of the target video, and plays the first video segment to the user through a video playback sub-interface in the user interface; When the first video segment is played, the user terminal collects the user's sight point on the user interface in real time, and determines the stop position of the user's sight point in several first segment images of the first video segment based on the position of the interface sight point and the position and area of ​​the video playback sub-interface in the user interface; The user terminal counts the duration of the sight line of the picture at each stop position, and sends the stop position and the stop duration of the sight line of the picture in each first segment picture to the second cloud server; The second cloud server determines the user's attention area and attention target in the first video segment based on the stay position and the stay duration of the screen sight point in each first segment screen, and determines whether the user's attention area belongs to the attention target; If the user's attention area belongs to the attention target, the second cloud server updates the rendering range in the initial 3D rendering effect parameters by using the area where the attention target is located to obtain the target 3D rendering effect parameters; if the attention area does not belong to the attention target, the second cloud server updates the rendering range in the initial 3D rendering effect parameters by using the attention area and the area where the attention target is located to obtain the target 3D rendering effect parameters; Alternatively, if the user's focus area belongs to the focus target, the second cloud server lowers the rendering accuracy of other non-target areas other than the focus target to obtain the target 3D rendering effect parameters; if the focus area does not belong to the focus target, the second cloud server lowers the rendering accuracy of non-focus areas other than the focus area and the non-target area to obtain the target 3D rendering effect parameters; The first cloud server performs screen rendering based on the target video data according to the target 3D rendering effect parameter to obtain a second video segment of the target video except the first video segment, and sends the second video segment to the user terminal, wherein the target video data is the video data of the target video except the at least part of the video data; After the first video segment is played, the user terminal continues to play the second video segment to the user, thereby achieving the playback of the target video.

2. The method according to claim 1, characterized in that Determining the user's focus area in the first video segment based on the stay position and stay duration of the sight point in each first segment screen includes: For each first segment picture, perform human body detection on the first segment picture using a human body detection model, and divide the first segment picture into human body areas and non-human body areas based on the human body detection result; Performing face detection on the human body region in the first image segment using a face detection model, and dividing the human body region into a face region and a non-face region based on the face detection result; Counting the dwell time of each segmented area where the sight point of the screen stays, and determining whether the dwell time corresponding to each segmented area exceeds a preset area time, wherein the segmented areas include the face area, the non-face area, and the non-human body area; Determine the segmented area whose corresponding area stay time exceeds the preset area time as the focus sub-area in the first segment picture; The user's focus area in the first video segment is determined based on the focus sub-area in each first segment screen.

3. The method according to claim 1, characterized in that Determining the user's attention target in the first video segment based on the stay position and stay duration of the sight point in each first segment screen includes: For each first segment picture, using a human body detection model to perform human body recognition on the first segment picture, and dividing the first segment picture into human body areas and non-human body areas based on the human body recognition results; Performing face recognition on the human body regions in the first image segment using a face recognition model, and determining the person indicated by each human body region based on the face recognition result; Counting the duration of a person's stay in each human body region at the position where the sight line of the screen falls, and determining whether the person corresponding to each human body region stays for more than a preset duration; Determine the person indicated by the human body region whose corresponding person stays for a time period exceeding the preset person stay time as a focus sub-target in the first segment picture; The focus target of the user in the first video segment is determined based on the focus sub-targets in each first segment screen.

4. The method according to claim 1, characterized in that The method of determining the stopping position of the user's sight line landing point in the first plurality of first segment images of the first video segment based on the position of the interface sight line landing point and according to the position and area of ​​the video playback sub-interface in the user interface includes: Determining an offset of the video playback sub-interface relative to the user interface according to a position of the video playback sub-interface in the user interface; Determining a scaling ratio of the video playback sub-interface relative to the user interface according to an area of ​​the video playback sub-interface in the user interface; For each first segment picture, based on the position of the interface sight point when playing the first segment picture, the stop position of the screen sight point in the first segment picture is determined according to the offset and the scaling ratio.

5. A video playback system based on 3D dynamic rendering, characterized in that: The system includes a user terminal, a first cloud server and a second cloud server; The first cloud server is configured to respond to a video playback instruction of the user terminal and send a video rendering file to the user terminal, wherein the video rendering file includes at least part of the video data of the target video indicated by the video playback instruction and initial 3D rendering effect parameters, and the initial 3D rendering effect parameters include a rendering range and a rendering accuracy; The user terminal is configured to perform picture rendering based on at least part of the video data according to the initial 3D rendering effect parameters to obtain a first video segment of the target video, and play the first video segment to the user through a video playback sub-interface in the user interface; The user terminal is used to collect the user's sight point on the user interface in real time when the first video segment is played, and determine the stop position of the user's sight point in the first segment screens of the first video segment based on the position of the interface sight point and the position and area of ​​the video playback sub-interface in the user interface; The user terminal is used to count the duration of the sight line of the picture at each stop position, and send the stop position and the stop duration of the sight line of the picture in each first segment picture to the second cloud server; The second cloud server is used to determine the user's attention area and attention target in the first video segment based on the stay position and the stay duration of the screen sight point in each first segment screen, and determine whether the user's attention area belongs to the attention target; The second cloud server is configured to update the rendering range in the initial 3D rendering effect parameters using the area where the target of interest is located to obtain the target 3D rendering effect parameters if the user's attention area belongs to the target of interest; and to update the rendering range in the initial 3D rendering effect parameters using the attention area and the area where the target of interest is located to obtain the target 3D rendering effect parameters if the attention area does not belong to the target of interest; Alternatively, the second cloud server is configured to, if the user's focus area belongs to the focus target, lower the rendering accuracy of other non-target areas other than the area where the focus target is located to obtain the target 3D rendering effect parameters; if the focus area does not belong to the focus target, lower the rendering accuracy of non-focus areas other than the focus area and the non-target area to obtain the target 3D rendering effect parameters; The first cloud server is configured to perform picture rendering based on the target video data according to the target 3D rendering effect parameter to obtain a second video segment of the target video except the first video segment, and send the second video segment to the user terminal, wherein the target video data is the video data of the target video except the at least part of the video data; The user terminal is used to continue playing the second video segment to the user after the first video segment is played, so as to achieve the playback of the target video.

6. The system according to claim 5, characterized in that Determining the user's focus area in the first video segment based on the stay position and stay duration of the sight point in each first segment screen includes: For each first segment picture, perform human body detection on the first segment picture using a human body detection model, and divide the first segment picture into human body areas and non-human body areas based on the human body detection result; Performing face detection on the human body region in the first image segment using a face detection model, and dividing the human body region into a face region and a non-face region based on the face detection result; Counting the dwell time of each segmented area where the sight point of the screen stays, and determining whether the dwell time corresponding to each segmented area exceeds a preset area time, wherein the segmented areas include the face area, the non-face area, and the non-human body area; Determine the segmented area whose corresponding area stay time exceeds the preset area time as the focus sub-area in the first segment picture; The user's focus area in the first video segment is determined based on the focus sub-area in each first segment screen.

7. The system according to claim 5, characterized in that Determining the user's attention target in the first video segment based on the stay position and stay duration of the sight point in each first segment screen includes: For each first segment picture, using a human body detection model to perform human body recognition on the first segment picture, and dividing the first segment picture into human body areas and non-human body areas based on the human body recognition results; Performing face recognition on the human body regions in the first image segment using a face recognition model, and determining the person indicated by each human body region based on the face recognition result; Counting the duration of a person's stay in each human body region at the position where the sight line of the screen falls, and determining whether the person corresponding to each human body region stays for more than a preset duration; Determine the person indicated by the human body region whose corresponding person stays for a time period exceeding the preset person stay time as a focus sub-target in the first segment picture; The focus target of the user in the first video segment is determined based on the focus sub-targets in each first segment screen.

8. The system according to claim 5, characterized in that The method of determining the stopping position of the user's sight line landing point in the first plurality of first segment images of the first video segment based on the position of the interface sight line landing point and according to the position and area of ​​the video playback sub-interface in the user interface includes: Determining an offset of the video playback sub-interface relative to the user interface according to a position of the video playback sub-interface in the user interface; Determining a scaling ratio of the video playback sub-interface relative to the user interface according to an area of ​​the video playback sub-interface in the user interface; For each first segment picture, based on the position of the interface sight point when playing the first segment picture, the stop position of the screen sight point in the first segment picture is determined according to the offset and the scaling ratio.

9. A computer device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the video playback method based on 3D dynamic rendering as described in any one of claims 1 to 4 are performed.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the video playback method based on 3D dynamic rendering as described in any one of claims 1 to 4 are executed.

Images