Video processing method and device, equipment, storage medium and program product

Abstract

The invention provides a video processing method and device, equipment, a storage medium and a program product. Comprising the following steps: based on eye movement data of a first object for a first video frame sequence in a target video and a second video frame sequence in the target video, determining gaze probability distribution of the first object on each region of the second video frame sequence; determining a subtitle priority of a subtitle text corresponding to the second video frame sequence; the subtitle priority is used for representing the importance degree of the subtitle text in the target video; performing video frame identification on each video frame in the second video frame sequence in sequence to obtain a contour area of each second object in the video frames; determining subtitle parameters of the subtitle text in the second video frame sequence based on the gaze probability distribution, the subtitle priority and the contour region; and performing subtitle rendering on the second video frame sequence based on the subtitle parameters. According to the invention, accurate layout and dynamic adaptation of the subtitle text in the target video can be realized.

Description

Technical Field

[0001] This application relates to the field of Internet technology, and in particular to a video processing method, apparatus, device, storage medium, and program product. Background Technology

[0002] With the explosive growth of video content, traditional static subtitle embedding methods can no longer meet users' demands for efficient information access and enhanced viewing experience. Especially in scenarios such as live streaming, education, and film and television, the real-time performance, accuracy, and visual adaptability of subtitles have become key challenges. Meanwhile, breakthroughs in artificial intelligence technology have provided new pathways for subtitle generation and layout optimization. For example, AI-powered subtitle functions help the hearing impaired achieve information accessibility, and smart obstacle avoidance glasses provide environmental perception support for the visually impaired. These applications demonstrate that intelligent subtitle systems combining computer vision, natural language processing, and reinforcement learning are becoming important tools for improving the interactivity and inclusivity of video content.

[0003] Related technologies generate subtitles using fixed rules, such as arranging text linearly at the bottom of the screen according to a timeline. While simple to implement, this technology struggles to adapt to dynamic scene changes. For instance, when the speaker moves or a new object enters the frame, the subtitles can easily overlap with the visual focus area, causing occlusion. Furthermore, some research on dynamic obstacle avoidance technologies attempts to adjust subtitle positions by incorporating visual information, such as vision-based dynamic obstacle avoidance algorithms in mobile robotics. These methods detect obstacle outlines and plan paths; however, they only address moving obstacles in physical space and do not consider the user's attention distribution or semantic scene understanding, resulting in a lack of human-like subtitle layout. Summary of the Invention

[0004] This application provides a video processing method, apparatus, device, storage medium, and program product that can achieve accurate layout and dynamic adaptation of subtitle text in a target video.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a video processing method, the method comprising: determining the gaze probability distribution of the first object in each region of the second video frame sequence based on eye-tracking data of a first object in a first video frame sequence in a target video and a second video frame sequence in the target video; the second video frame sequence being an adjacent video frame sequence following the first video frame sequence; determining the subtitle priority of subtitle text corresponding to the second video frame sequence; the subtitle priority being used to characterize the importance of the subtitle text in the target video; sequentially performing video frame recognition for each video frame in the second video frame sequence to obtain the contour region of each second object in the video frame; determining subtitle parameters of the subtitle text in the second video frame sequence based on the gaze probability distribution, the subtitle priority, and the contour region; and rendering subtitles for the second video frame sequence based on the subtitle parameters.

[0007] This application provides a video processing apparatus, comprising: a first determining module, configured to determine the gaze probability distribution of the first object in each region of the second video frame sequence based on eye-tracking data of a first object in a first video frame sequence in a target video and a second video frame sequence in the target video; the second video frame sequence is an adjacent video frame sequence following the first video frame sequence; a second determining module, configured to determine the subtitle priority of subtitle text corresponding to the second video frame sequence; the subtitle priority is used to characterize the importance of the subtitle text in the target video; a video frame recognition module, configured to sequentially perform video frame recognition for each video frame in the second video frame sequence to obtain the contour region of each second object in the video frame; a third determining module, configured to determine the subtitle parameters of the subtitle text in the second video frame sequence based on the gaze probability distribution, the subtitle priority, and the contour region; and a subtitle rendering module, configured to render subtitles in the second video frame sequence based on the subtitle parameters.

[0008] In the above scheme, the eye-tracking data includes a fixation point sequence and a pupil diameter sequence; the first determining module is further configured to: encode the fixation point sequence to obtain the fixation point features of the first object in relation to the first video frame sequence; determine the pupil diameter change rate of the first object in relation to the first video frame sequence based on the pupil diameter sequence; extract spatiotemporal features from the second video frame sequence to obtain the spatiotemporal features of the second video frame sequence; and determine the fixation probability distribution based on the fixation point features, the pupil diameter change rate, and the spatiotemporal features.

[0009] In the above scheme, the first determining module is further configured to: perform spatiotemporal segmentation processing on the second video frame sequence to obtain spatiotemporal blocks corresponding to the second video frame sequence; extract time features from the spatiotemporal blocks to obtain time features of the second video frame sequence; the time features are used to characterize the dynamic change information between consecutive video frames in the second video frame sequence; extract spatial features from the spatiotemporal blocks to obtain spatial features of the second video frame sequence; the spatial features are used to characterize the static visual information in each video frame in the second video frame sequence; and perform a first weighted fusion on the time features and the spatial features to obtain the spatiotemporal features.

[0010] In the above scheme, the first determining module is further configured to: perform feature mapping on the pupil diameter change rate to obtain attention weights; perform weighted processing on the fixation point features based on the attention weights to obtain weighted features; perform an outer product operation on the spatiotemporal features and the weighted features to obtain a spatial attention matrix; and determine the fixation probability distribution based on the spatial attention matrix and the spatiotemporal features.

[0011] In the above scheme, the first determining module is further configured to: perform a second weighted fusion on the spatial attention matrix and the spatiotemporal features to obtain fused features; perform upsampling processing on the fused features to obtain upsampled features; and perform feature mapping on the upsampled features to obtain the gaze probability distribution.

[0012] In the above scheme, the device further includes a cropping module, which is used to: determine the pupil diameter change rate of the first object in relation to the first video frame sequence; determine the sliding window length based on the pupil diameter change rate; and crop video frames starting from the terminating video frame in the first video frame sequence in the target video based on the sliding window length to obtain the second video frame sequence.

[0013] In the above scheme, the second determining module is further configured to: encode the subtitle text corresponding to the second video frame sequence to obtain the text encoding vector of the subtitle text; perform semantic parsing on the text encoding vector to obtain the part-of-speech probability and sentiment intensity value of each word in the subtitle text; and determine the subtitle priority of the subtitle text based on the part-of-speech probability and the sentiment intensity value.

[0014] In the above scheme, the second determining module is further configured to: determine the part-of-speech of each word in the subtitle text based on the part-of-speech probability, and determine the sentiment attribute of each word in the subtitle text based on the sentiment intensity value; obtain a first weight value of the part-of-speech and a second weight value of the sentiment attribute; and use the first weight value and the second weight value to perform a weighted summation of the part-of-speech probability and the sentiment intensity to obtain the subtitle priority.

[0015] In the above scheme, the video frame recognition module is further configured to: perform target detection sequentially on each video frame in the second video frame sequence to obtain the bounding box of each second object in the video frame and the semantic label of the bounding box; determine the segmentation threshold corresponding to the bounding box based on the semantic label; and perform image segmentation on the image region corresponding to the bounding box based on the segmentation threshold to obtain the contour region of each second object in the video frame.

[0016] In the above scheme, the third determining module is further configured to: for each video frame in the second video frame sequence, determine a set of candidate subtitle parameters for the subtitle text in the video frame; and, based on a preset objective function, determine the subtitle parameters of the subtitle text in the video frame from the set of candidate subtitle parameters; the objective function includes: a first penalty term negatively correlated with the occlusion area, a reward term positively correlated with the subtitle readability, and a second penalty term negatively correlated with the difference; wherein the occlusion area is determined based on the contour region, the subtitle readability is determined based on the gaze probability distribution and the subtitle priority, and the difference is determined based on the subtitle parameters of the video frame and the adjacent video frames preceding the video.

[0017] In the above scheme, the third determining module is further configured to: determine the target value of each candidate subtitle parameter in the candidate subtitle parameter set under the objective function; and determine the candidate subtitle parameter with the maximum target value as the subtitle parameter of the subtitle text in the video frame.

[0018] In the above scheme, the third determining module is further configured to: determine the candidate subtitle size, candidate transparency, candidate subtitle region, and candidate center point coordinates of the subtitle text in the video frame based on the candidate subtitle parameters; determine the visual focus coordinates of the first object in the video frame based on the gaze probability distribution; determine a first penalty value based on the candidate subtitle region, the contour region, and the first penalty item; determine a reward value based on the candidate subtitle size, the candidate transparency, the subtitle priority, the coordinate distance between the visual focus coordinates and the candidate center point coordinates, and the reward item; determine a second penalty value based on the candidate center point coordinates, the center point coordinates of the subtitle text in adjacent video frames, and the second penalty item; and determine the target value based on the first penalty value, the reward value, and the second penalty value.

[0019] In the above scheme, the third determining module is further configured to: obtain the third weight value of the first penalty value, the fourth weight value of the reward value, and the fifth weight value of the second penalty value; and use the third weight value, the fourth weight value, and the fifth weight value to perform a weighted summation on the first penalty value, the reward value, and the second penalty value to obtain the target value.

[0020] This application provides an electronic device, including: a memory for storing computer-executable instructions; and a processor for executing the computer-executable instructions stored in the memory to implement the video processing method provided in this application.

[0021] This application provides a computer-readable storage medium storing computer-executable instructions for implementing the video processing method provided in this application when executed by a processor.

[0022] This application provides a computer program product, which includes computer-executable instructions stored in a computer-readable storage medium. When the processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, it implements the video processing method provided in this application.

[0023] The embodiments of this application have the following beneficial effects:

[0024] During video processing, by collecting eye-tracking data of the first object watching the first video frame sequence and combining it with adjacent second video frame sequences, the gaze probability distribution of the first object in each region of the second video frame sequence can be accurately predicted. Simultaneously, the subtitle priority of the corresponding subtitle text in the second video frame sequence is determined to differentiate the importance of information. Then, the contour region of the second object in each video frame is extracted through video frame recognition. Finally, based on the gaze probability distribution, subtitle priority, and contour region, subtitle parameters are determined and subtitle rendering is completed. Thus, by using eye-tracking data to predict the gaze probability distribution, the subtitle layout can conform to the user's visual habits, reducing interference with the user's focus area and improving viewing smoothness. Adjusting the presentation method according to subtitle priority makes important information in the subtitles more prominent, improving the effectiveness of information delivery. Dynamic obstacle avoidance of subtitles is achieved by recognizing the contour region of the second object, avoiding obscuring key video content and ensuring content integrity; and frame-by-frame processing of the second video frame sequence ensures that subtitle parameters dynamically adapt to the screen, maintaining the continuity and naturalness of the presentation. In summary, by combining information such as gaze probability distribution, subtitle priority, and the outline region of the second object, it is possible to achieve precise layout and dynamic adaptation of subtitle text in the target video, which not only improves the user viewing experience but also ensures the effectiveness of information delivery and the integrity of the content. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the architecture of the video processing system 100 provided in this application embodiment;

[0026] Figure 2 This is a schematic diagram of the structure of the electronic device 40 provided in the embodiment of this application;

[0027] Figure 3 This is an optional flowchart illustrating the video processing method provided in an embodiment of this application;

[0028] Figure 4 This is another optional flowchart illustrating the video processing method provided in the embodiments of this application;

[0029] Figure 5 This is a schematic diagram illustrating the implementation process of determining spatiotemporal features provided in an embodiment of this application;

[0030] Figure 6 This is a schematic diagram of the implementation process for determining the gaze probability distribution provided in an embodiment of this application;

[0031] Figure 7 This is a schematic diagram illustrating the implementation process of determining the second video frame sequence provided in an embodiment of this application;

[0032] Figure 8 This is a schematic diagram illustrating the implementation process of determining subtitle parameters provided in an embodiment of this application;

[0033] Figure 9 This is a schematic diagram of the implementation process for determining the target value provided in the embodiments of this application. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0035] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments, but it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0036] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0037] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0038] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0039] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0040] The relevant technologies have several limitations in video processing: First, they lack real-time performance. Relying on offline processing or fixed time window analysis, they struggle to handle rapidly changing scenarios such as camera cuts or character movements. For example, when the user's visual focus suddenly shifts, the subtitles cannot adjust their position in time, resulting in key information being obscured. Second, they lack multimodal data fusion. These technologies rarely integrate multi-source data such as video, audio, and eye tracking. For instance, relying solely on speech recognition to generate subtitles without considering the user's gaze distribution to optimize display position leads to a high probability of overlap between subtitles and the visual focus area. Third, they lack contextual understanding. These technologies generally ignore the connection between scene semantics and user attention. For example, they rely solely on object detection for obstacle avoidance without distinguishing the priority of "faces" versus "background text," potentially causing subtitles to avoid secondary objects while obscuring the face of a key character. Fourth, they have poor dynamic adaptability. These technologies use preset rules (e.g., fixed safety distances) and cannot dynamically adjust strategies based on content semantics and user attention. For example, the display methods for high-priority and low-priority subtitles are indistinguishable, affecting the efficiency of subtitle information delivery.

[0041] To address at least one of the problems in the aforementioned related technologies, the embodiments of this application break through the limitations of static rules, integrate visual focus detection technology to accurately capture user attention; combine scene semantic understanding to deeply analyze the content of the screen and the meaning of the subtitles; and combine real-time decision-making algorithms to drive intelligent dynamic obstacle avoidance and personalized layout of the subtitles, thereby enhancing the efficiency of subtitle information transmission and improving the user viewing experience.

[0042] The following describes exemplary applications of the video processing device (i.e., electronic device) provided in the embodiments of this application. The video processing device provided in the embodiments of this application can be implemented as various types of terminals capable of video processing, such as laptops, tablets, desktop computers, set-top boxes, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices, smartphones, smart speakers, smartwatches, smart TVs, and in-vehicle terminals. It can also be implemented as a server. The following will describe exemplary applications when the video processing device is implemented as a server.

[0043] See Figure 1 , Figure 1 This is a schematic diagram of the architecture of the video processing system 100 provided in this application embodiment. In order to support a video processing application, the terminal 400 connects to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0044] Terminal 400 sends a video processing request to server 200, which constitutes the video processing device of this embodiment. In response to the video processing request, server 200 determines the subtitle parameters of the subtitle text in the second video frame sequence using the video processing method provided in this embodiment. Finally, it renders the subtitles in the second video frame sequence based on the subtitle parameters to obtain a subtitle rendering result. The subtitle rendering result is then returned to terminal 400 for display on terminal 400.

[0045] In some embodiments, server 200 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminals and servers can be connected directly or indirectly via wired or wireless communication, which is not limited in this embodiment.

[0046] See Figure 2 , Figure 2 This is a schematic diagram of the structure of the electronic device 40 provided in the embodiment of this application. Figure 2 The illustrated electronic device 40 may be a video processing device. The electronic device includes at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. The various components in the electronic device are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 2 The general labeled all buses as Bus System 440.

[0047] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0048] User interface 430 includes one or more output devices 431 that enable the presentation of media content, including one or more speakers and one or more visual displays. User interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0049] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 410.

[0050] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.

[0051] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0052] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0053] The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including Bluetooth, WiFi, and Universal Serial Bus (USB); the presentation module 453 is used to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 431 associated with the user interface 430 (e.g., a display screen, a speaker, etc.); the input processing module 454 is used to detect and translate one or more user inputs or interactions from one or more input devices 432.

[0054] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2A video processing device 455 stored in memory 450 is shown. This device can be software in the form of programs and plugins, and includes the following software modules: a first determining module 4551, a second determining module 4552, a video frame recognition module 4553, a third determining module 4554, and a subtitle rendering module 4555. These modules are logically connected and can therefore be arbitrarily combined or further separated according to their implemented functions. The functions of each module will be described below.

[0055] In some other embodiments, the apparatus provided in this application can be implemented in hardware. For example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the video processing method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0056] In some embodiments, the terminal or server can implement the video processing method provided in this application by running various computer-executable instructions or computer programs. For example, computer-executable instructions can be microprogram-level commands, machine instructions, or software instructions. Computer programs can be native programs or software modules in an operating system; they can be native applications (APPs), i.e., programs that need to be installed in the operating system to run, or they can be applets that can be embedded in any APP, i.e., programs that only need to be downloaded to a browser environment to run. In summary, the aforementioned computer-executable instructions can be any form of instruction, and the aforementioned computer programs can be any form of application, module, or plugin.

[0057] The video processing methods provided in the embodiments of this application can be executed by an electronic device, which can be a server or a terminal. That is, the video processing methods in the embodiments of this application can be executed by a server, by a terminal, or by interaction between a server and a terminal.

[0058] See Figure 3 , Figure 3 This is an optional flowchart illustrating the video processing method provided in this application embodiment, which will be combined with... Figure 3 The steps shown are explained below, taking a server as the execution subject of the video processing method as an example. The method includes the following steps S101 to S105:

[0059] Step S101: Based on the eye-tracking data of the first object in the first video frame sequence in the target video and the second video frame sequence in the target video, determine the gaze probability distribution of the first object in each region of the second video frame sequence.

[0060] In this embodiment, the first object refers to the subject watching the target video, such as a user on a video playback platform. The target video refers to the complete video content watched by the first object, which includes a continuous sequence of video frames, audio information, and potential subtitle requirements. The first video frame sequence refers to the set of consecutive video frames at the beginning of the target video. Eye-tracking data refers to the eye movements and physiological characteristics recorded by an eye-tracking device (e.g., an eye tracker or a head-mounted eye-tracking sensor) when the first object watches the first video frame sequence. Eye-tracking data can be used to reflect the attention state of the first object. For example, eye-tracking data can be pupil size data changing over time and eye gaze position coordinates. The second video frame sequence is the adjacent video frame sequence following the first video frame sequence. The second video frame sequence is the target window for which the gaze probability distribution needs to be predicted. The correlation between the content of the second video frame sequence and the first video frame sequence affects the accuracy of attention prediction. The gaze probability distribution refers to the probability that each area in the screen is gazed upon when the first object watches the second video frame sequence, usually represented by a gaze probability value of 0-1. The higher the gaze probability value, the more likely the area is to be focused upon. Region segmentation can be performed by pixels, grids, or semantics. The gaze probability distribution is a prospective prediction of attention in the second video frame sequence based on eye-tracking data from the first video frame sequence. It is used to guide subtitles to avoid high-probability gaze regions, thereby preventing occlusion of video content that the user is focused on.

[0061] For example, suppose a user is watching a food preparation video. The first video frame sequence might be the first few seconds of the video, showing a chef cutting tomatoes, continuously demonstrating the process from whole to diced tomatoes, corresponding to multiple consecutive frames. During this time, an eye-tracking device records that the user's eyes are consistently focused on the point where the knife touches the tomato, with the pupils slightly constricting due to focus. This data is the eye-tracking data corresponding to the first video frame sequence. The second video frame sequence is the next few seconds following the first sequence, showing the chef pouring tomato chunks into the pan. This part is the target window for predicting the user's attention. Based on the eye-tracking data of the user focusing on the cutting action in the first video frame sequence, the distribution of the user's attention in the second video frame sequence is predicted. For example, the edge of the pan has the highest probability of being looked at (0.9), the adjacent stove area has a lower probability of being looked at (0.3), and the condiment bottle in the corner of the screen has the lowest probability of being looked at (0.1).

[0062] Step S102: Determine the subtitle priority of the subtitle text corresponding to the second video frame sequence.

[0063] In this embodiment, subtitle text refers to text information synchronized with the content of the second video frame sequence, used to assist the first object in understanding the video content, such as narration, dialogue, or knowledge point annotations. Subtitle priority is used to characterize the importance of the subtitle text in the target video. A higher subtitle priority requires greater readability, for example, larger font size or a more prominent position. Subtitle priority is typically determined based on the semantic importance of the subtitle text. For example, in a video clip of the moon's rotation, the subtitle "Tidal locking is the reason why the moon always faces the earth with the same side" contains a core knowledge point, and its priority can be set to 0.9; while the subtitle "In the picture: craters on the surface of the moon" is supplementary information, and its priority can be set to 0.3.

[0064] Step S103: Perform video frame recognition on each video frame in the second video frame sequence in sequence to obtain the contour region of each second object in the video frame.

[0065] In this embodiment, video frame recognition refers to the process of analyzing and identifying objects and scenes in each video frame of a second video frame sequence using computer vision algorithms. The core purpose of video frame recognition is to locate important objects in the video frames, i.e., the second objects, to provide a basis for avoiding subtitle occlusion later. The second object refers to an important visual object in the video frame that needs to be avoided from being occluded by subtitles. The importance of the second object can be determined by the content of the video frame. For example, the second object can be the main object, a person, or a key symbol in the video frame. The contour region refers to the edge contour range of the second object obtained through video frame recognition. The contour region can be represented by a pixel-level mask or polygon vertex coordinates, and is used to precisely define the spatial range of the second object. For example, the contour region of the moon model in the video frame can be represented by a set of polygon vertex coordinates: (350, 280), (420, 280), (420, 350), (350, 350). This contour region defines the circular range of the moon model in the video frame.

[0066] In some embodiments, for each video frame in the second video frame sequence, a target detection process can be used to output the bounding box and semantic label of each second object in the video frame; then, based on the semantic label of the bounding box, a segmentation threshold is determined for image segmentation of the image region corresponding to the bounding box of each second object; finally, based on the segmentation threshold, the contour region of the second object is segmented from the image region corresponding to the bounding box.

[0067] Step S104: Based on the gaze probability distribution, subtitle priority, and contour region, determine the subtitle parameters of the subtitle text in the second video frame sequence.

[0068] In this embodiment, the subtitle parameters refer to the specific attributes of the subtitle text in the video frame, which are used to control the subtitle position, font size and transparency of the subtitle text, so as to balance the three major objectives of subtitle readability, not obscuring the second object and conforming to the gaze probability distribution.

[0069] In some embodiments, after obtaining the gaze probability distribution, subtitle priority, and contour region, the subtitle parameters of the subtitle text in the second video frame sequence can be determined based on the gaze probability distribution, subtitle priority, and contour region, so as to maximize the subtitle readability of the subtitle text in the second video frame sequence, minimize the occlusion area with the second object, and minimize the position jump frequency between adjacent video frames.

[0070] Step S105: Render subtitles for the second video frame sequence based on subtitle parameters.

[0071] In this embodiment of the application, subtitle rendering refers to the process of adding subtitle text to the second video frame sequence according to determined subtitle parameters, and using video processing technology to make the subtitle text appear on the screen, and finally outputting a video frame sequence with optimized subtitles.

[0072] In some embodiments, after rendering subtitles on the second video frame sequence based on subtitle parameters, a subtitle rendering result can be obtained. The subtitle rendering result refers to the second video frame sequence after subtitle overlay is completed after subtitle parameter optimization. Its characteristic is that the subtitle text avoids the outline area and high gaze probability area of ​​the second object, while ensuring the readability of high-priority subtitles, thus achieving a balance between information delivery and viewing experience.

[0073] In some embodiments, subtitle rendering of the second video frame sequence based on subtitle parameters can be achieved in the following way: First, parse the subtitle parameters to extract the subtitle coordinates, font size, transparency, and corresponding subtitle text of each video frame; then call the OpenCV putText function to draw the subtitle at the specified position of the video frame according to the parameters, and process the superposition of text and background through a transparency blending algorithm; finally, stitch all the rendered video frames together in chronological order, synchronize the audio track, and output a video clip with subtitle text.

[0074] In some embodiments, when the first video frame sequence is the first video frame sequence in the target video, since the historical eye movement data of the first object cannot be obtained, the first video frame sequence may not consider the gaze probability distribution of the first object, but only consider the subtitle priority and the outline region of the second object to set the subtitle parameters and render the subtitle.

[0075] In some embodiments, after rendering subtitles on the second video frame sequence based on subtitle parameters, the second video frame sequence can be determined as a new first video frame sequence, and a video frame sequence extracted from the terminating video frame of the second video frame sequence can be determined as a new second video frame sequence. New eye-tracking data is recorded when the first object views the second video frame sequence. Then, subtitle parameters for the new second video frame sequence are determined based on the new first video frame sequence and the new eye-tracking data, and subtitle rendering is completed. This process is repeated until all video frames in the target video have been subtitled, and the target video with subtitle text is output.

[0076] The video processing method provided in this application, during video processing, collects eye-tracking data of a first object watching a first video frame sequence and combines it with adjacent second video frame sequences to accurately predict the gaze probability distribution of the first object in each region of the second video frame sequence. Simultaneously, it determines the subtitle priority of the corresponding subtitle text in the second video frame sequence to differentiate the importance of information. Then, it extracts the contour region of the second object in each video frame through video frame recognition, and finally determines the subtitle parameters and completes subtitle rendering based on the gaze probability distribution, subtitle priority, and contour region. Thus, by using eye-tracking data to predict the gaze probability distribution, the subtitle layout can conform to the user's visual habits, reduce interference with the user's focus area, and improve viewing smoothness. Adjusting the presentation method according to subtitle priority makes important information in the subtitles more prominent, improving the effectiveness of information transmission. Dynamic obstacle avoidance of subtitles is achieved by recognizing the contour region of the second object, avoiding obscuring key video content and ensuring content integrity; and frame-by-frame processing of the second video frame sequence ensures that subtitle parameters dynamically adapt to the screen, maintaining the continuity and naturalness of the presentation. In summary, by combining information such as gaze probability distribution, subtitle priority, and the outline region of the second object, it is possible to achieve precise layout and dynamic adaptation of subtitle text in the target video, which not only improves the user viewing experience but also ensures the effectiveness of information delivery and the integrity of the content.

[0077] The following describes the interaction between the terminal and the server in the video processing system, illustrating the video processing method in this embodiment. It should be noted that this video processing method is implemented through interaction between the terminal and the server, and is essentially the same as the video processing method executed by the server in the above embodiments. The only difference is that this embodiment also describes the actions performed by the terminal during the execution of the video processing method. Furthermore, some steps can be executed by either the terminal or the server. Therefore, for steps in this embodiment that are identical in content to those in the above embodiments but have different execution entities, this embodiment is merely illustrative. In practice, any execution entity can perform these steps, and this embodiment does not limit this.

[0078] Figure 4 This is another optional flowchart illustrating the video processing method provided in the embodiments of this application, such as... Figure 4 As shown, the method includes the following steps S201 to S218:

[0079] Step S201: The terminal receives video processing operations input by the user.

[0080] In this embodiment of the application, a video processing application may be running on the terminal. Users can perform video processing operations on the client side of the video processing application. The video processing application can provide video processing functions, and users can perform video processing operations on the video processing function page to trigger video processing requests.

[0081] In some embodiments, when a user inputs a video processing operation, they can also simultaneously input a target video. When the terminal receives the target video, a confirmation window will pop up on the video processing function page. After the terminal detects that the user has clicked the confirmation button, further processing of the target video will be performed. Alternatively, in other embodiments, the user can directly input the target video on the video processing function page. Upon receiving the target video, the terminal can directly trigger the video processing function to further process the image to be detected.

[0082] In step S202, the terminal generates a video processing request in response to the video processing operation.

[0083] In this embodiment, user-input data can be encapsulated into a video processing request. For example, the target video is displayed on the screen of a video processing application. The user can select or confirm the target video on the screen according to actual needs, and then the user-input target video can be encapsulated into a video processing request.

[0084] In step S203, the terminal sends a video processing request to the server.

[0085] In step S204, the server responds to the video processing request by encoding the gaze point sequence to obtain the gaze point features of the first object for the first video frame sequence.

[0086] In this embodiment, the eye-tracking data of the first object for the first video frame sequence in the target video includes a gaze point sequence. A gaze point sequence refers to the continuous sequence of coordinates of the eyeball's gaze position over time or video frames recorded in the eye-tracking data when the first object views the first video frame sequence. Each element corresponds to the gaze point pixel coordinates (x, y) at a certain moment or video frame, reflecting the change in the position of the gaze. For example, for a first video frame sequence containing 30 frames in the past second, the recorded gaze point sequence could be [(320, 240), (322, 239), ..., (318, 242)], indicating that the first object's gaze is mainly concentrated in the central area of ​​the image. Encoding processing refers to the process of converting the unstructured gaze point sequence into structured features that can be processed by algorithms or models. The purpose of encoding processing is to extract key information from the gaze point sequence. Gazing point features refer to the structured data obtained after encoding processing that can characterize the core patterns of the gaze point sequence. Gazing point features can include information such as gaze position distribution, gaze frequency, and temporal change trends.

[0087] In some embodiments, the server responds to a video processing request by directly parsing the request to obtain the target video. First, a predetermined number of video frames can be extracted from the starting frame of the target video, and these frames are defined as a first video frame sequence. Subtitles are rendered for the corresponding subtitle text using a preset subtitle rule. Simultaneously, eye-tracking data of a first object is collected using an eye-tracking device for the first video frame sequence. A second video frame sequence is then extracted from the ending frame of the first video frame sequence, providing foundational data for subsequently determining the gaze probability distribution.

[0088] In some embodiments, after obtaining the gaze point sequence from the eye-tracking data, the gaze point sequence is encoded to obtain gaze point features. The encoding process can be implemented by inputting the gaze point sequence into a neural network capable of processing temporal data, such as a Long Short-Term Memory network or a gated recurrent unit. The gaze point sequence is then encoded using the neural network to obtain the gaze point features.

[0089] In step S205, the server determines the rate of change of the pupil diameter of the first object relative to the first video frame sequence based on the pupil diameter sequence.

[0090] In this embodiment, the eye-tracking data of the first object for the first video frame sequence in the target video also includes a pupil diameter sequence. The pupil diameter sequence refers to the continuous measurement sequence of pupil diameter changes over time as the first object watches the first video frame sequence, reflecting the intensity of attention. Generally, a larger pupil diameter indicates a more focused attention on the first object. For example, if the pupil diameter is recorded every 100ms during the playback of the first video frame sequence, the resulting pupil diameter sequence could be [3.2, 3.3, 3.5, 3.4, ..., 3.6], indicating that the pupil gradually dilates as the video content changes, and the first object's attention gradually becomes more focused.

[0091] The pupil diameter change rate is the ratio of the diameter difference between two adjacent measurement points in a pupil diameter sequence to the time interval, reflecting the speed of change in pupil size. A positive pupil diameter change rate indicates pupil dilation, while a negative rate indicates pupil constriction. The pupil diameter change rate indirectly reflects the drastic change in attention. For example, if the pupil diameter changes from 3.2 mm at t1 = 0.1 s to 3.5 mm at t2 = 0.2 s, with a time interval of 0.1 s, then the pupil diameter change rate is (3.5 - 3.2) / 0.1 = 3 mm / s, indicating rapid pupil dilation and a significant increase in attention.

[0092] In some embodiments, after obtaining the pupil diameter sequence in the eye-tracking data, the pupil diameter change rate can be determined based on the pupil diameter sequence. The determination of the pupil diameter change rate can be achieved as follows: First, calculate the pupil diameter difference between adjacent measurement points, such as the pupil diameter difference Δd = d(t+1) - d(t) between the pupil diameter d(t) of the t-th measurement point and the pupil diameter d(t+1) of the (t+1)-th measurement point; then, by dividing the pupil diameter difference Δd by the time interval between adjacent measurement points, the pupil diameter change rate synchronized with the first video frame sequence is obtained.

[0093] Step S206: The server extracts spatiotemporal features from the second video frame sequence to obtain the spatiotemporal features of the second video frame sequence.

[0094] In this embodiment, spatiotemporal feature extraction refers to the process of simultaneously extracting spatial and temporal features from a second video frame sequence. The purpose of spatiotemporal feature extraction is to capture the dynamic patterns of the video content in the second video frame sequence. Spatiotemporal features refer to structured data obtained through spatiotemporal feature extraction that can characterize the spatial distribution and temporal dynamics of the second video frame sequence. Spatiotemporal features include the position or shape of objects within a single frame, as well as information such as the direction of object movement or scene continuity between frames.

[0095] In some embodiments, see Figure 5 , Figure 5The step S206 shows that the server extracts spatiotemporal features from the second video frame sequence to obtain the spatiotemporal features of the second video frame sequence. This can be achieved by executing the following steps S2061 to S2064:

[0096] Step S2061: Perform spatiotemporal segmentation on the second video frame sequence to obtain the spatiotemporal block corresponding to the second video frame sequence.

[0097] In this embodiment, spatiotemporal segmentation refers to the process of dividing a continuous second video frame sequence into multiple local blocks in both spatial and temporal dimensions. The purpose of spatiotemporal segmentation is to simultaneously capture the spatial information and temporal dynamics of a local region. A spatiotemporal block is the result of spatiotemporal segmentation; it is a three-dimensional data block containing a local spatial region and continuous temporal frames. For example, approximately 0.17 seconds of continuous 5-frame video can be divided into multiple overlapping blocks with a spatial size of 16×16 pixels and a time window of 5 frames, resulting in a 16×16×5 spatiotemporal block.

[0098] In some embodiments, the server performs spatiotemporal segmentation on the second video frame sequence to obtain the spatiotemporal block corresponding to the second video frame sequence. This can be achieved in the following way: First, determine the time window length of the second video frame sequence, and then divide each video frame in the second video frame sequence into a grid according to a preset spatial size; then, extract a three-dimensional data block of a preset spatial size × time window length from multiple consecutive video frames of the second video frame sequence in a sliding manner to obtain the spatiotemporal block corresponding to the second video frame sequence.

[0099] Step S2062: Extract time features from the spatiotemporal block to obtain the time features of the second video frame sequence.

[0100] In this embodiment, temporal feature extraction refers to the process of extracting features representing dynamic changes from the temporal dimension of a spatiotemporal block, focusing on motion change information between frames. Temporal features are the result of temporal feature extraction and are used to characterize the dynamic change information between consecutive video frames in the second video frame sequence. For example, temporal features could be features such as the offset of a person's position and the amplitude of their movement in adjacent frames when a person is walking in a video.

[0101] In some embodiments, the server extracts temporal features from the spatiotemporal blocks to obtain the temporal features of the second video frame sequence. This can be achieved in the following way: First, for each spatiotemporal block, a convolutional kernel of a preset dimension (e.g., a 1×1×3 convolutional kernel) is slid in the temporal direction. The convolutional kernel sequentially covers the same spatial position of multiple consecutive frames in the spatiotemporal block. Local dynamic information, i.e., the convolution result, is obtained by calculating the changes in pixel values. Then, all convolution results are integrated to obtain the temporal features of the second video frame sequence.

[0102] Step S2063: Spatial features are extracted from the spatiotemporal block to obtain the spatial features of the second video frame sequence.

[0103] In this embodiment, spatial feature extraction refers to the process of extracting features representing static visual information from the spatial dimension of a spatiotemporal block, focusing on static visual information such as texture, shape, and color within a single frame. Spatial features are the result of spatial feature extraction and are used to represent the static visual information in each video frame of the second video frame sequence. For example, spatial features may be features such as the outline of a building, the texture of leaves, or the color distribution of clothing in a single frame.

[0104] In some embodiments, the server extracts spatial features from the spatiotemporal blocks to obtain the spatial features of the second video frame sequence. This can be achieved in the following way: First, for each frame in each spatiotemporal block, a convolution kernel of a preset dimension (e.g., a 3×3×1 convolution kernel) is slid in the spatial direction. The convolution kernel sequentially covers local sub-regions of a preset spatial size region within a single frame. By calculating pixel value differences, information such as texture and edges is captured, i.e., the convolution result. Then, all convolution results are integrated to obtain the spatial features of the second video frame sequence.

[0105] Step S2064: Perform a first weighted fusion of the temporal and spatial features to obtain the spatiotemporal features.

[0106] In this embodiment, the spatiotemporal feature is a comprehensive feature obtained by fusing temporal and spatial features, and it includes both dynamic change information and static visual information. The first weighted fusion refers to the process of fusing the temporal and spatial features by assigning different weights to them according to their importance. The weights can be dynamically adjusted to highlight more critical features.

[0107] In some embodiments, the server performs a first weighted fusion of temporal and spatial features to obtain spatiotemporal features, which can be achieved in the following way: First, the temporal and spatial features are input into the channel attention gating, and the importance weights of the temporal and spatial features are calculated through a learning mechanism. For example, the weight of temporal features is higher in dynamic scenes and the weight of spatial features is higher in static scenes. Then, the temporal and spatial features are weighted and summed according to their weights to obtain the fused spatiotemporal features.

[0108] Here, spatiotemporal segmentation is used to simultaneously capture spatial and temporal information of local areas. Combined with temporal and spatial feature extraction, dynamic changes and static visual information are accurately captured respectively. A first weighted fusion integrates these two types of features to obtain more comprehensive spatiotemporal features. This process effectively preserves key dynamic and static information in the second video sequence, providing high-quality feature support for subsequent steps such as determining the gaze probability distribution, thus improving the accuracy and reliability of video content-based analysis and decision-making.

[0109] In step S207, the server determines the gaze probability distribution based on gaze point features, pupil diameter change rate, and spatiotemporal features.

[0110] In some embodiments, see Figure 6 , Figure 6 The following steps illustrate how, in step S207, the server determines the gaze probability distribution based on gaze point features, pupil diameter change rate, and spatiotemporal features. This can be achieved by executing steps S2071 to S2074:

[0111] Step S2071: Perform feature mapping on the pupil diameter change rate to obtain attention weights.

[0112] In this embodiment, feature mapping is the process of converting the raw physiological signal of pupil diameter change rate into a weight value that can be used for attention calculation. A pre-defined function or model establishes the correspondence between the two. Attention weight is a numerical value used to characterize the degree of influence of pupil diameter change on fixation point features. The more drastic the pupil diameter change, the higher the attention weight, meaning that the fixation point at that moment is more worthy of attention. For example, when the video scene suddenly changes, the pupil diameter change rate is large, and the attention weight obtained after feature mapping can be 0.8; when the scene is stable, the pupil diameter change rate is small, and the attention weight obtained after feature mapping can be 0.2.

[0113] In some embodiments, the pupil diameter change rate is feature-mapped to obtain attention weights, which can be achieved as follows: First, a mapping function (e.g., a sigmoid function or a linear function) is determined, with the pupil diameter change rate as the input and the output range limited to 0-1; then, the pupil diameter change rates extracted from the first video frame sequence are input into the mapping function one by one to finally obtain the attention weights corresponding to each time step.

[0114] Step S2072: Based on attention weights, the fixation features are weighted to obtain weighted features.

[0115] In this embodiment, weighted processing refers to the process of scaling and adjusting the gaze point features using attention weights to highlight the gaze point features corresponding to high weights. Weighted features refer to the gaze point features after weighted processing; the weighted features retain the distribution trend of the original features while strengthening the regions corresponding to high attention.

[0116] In some embodiments, the gaze features are weighted based on attention weights to obtain weighted features. This can be achieved as follows: First, obtain the gaze features and their corresponding attention weights; then, multiply each element of the gaze features by the attention weight. For example, if the feature value of the upper right corner region in the gaze features is 0.3 and the attention weight is 0.8, then the weighted feature value of that region is 0.3 × 0.8 = 0.24. Repeat this operation to obtain the complete weighted feature vector.

[0117] Step S2073: Perform an outer product operation on the spatiotemporal features and weighted features to obtain the spatial attention matrix.

[0118] In this embodiment, the spatial attention matrix is ​​a matrix obtained by outer product operation. Each element in the spatial attention matrix is ​​the product of the corresponding elements of the spatiotemporal feature and the weighted feature. The spatial attention matrix is ​​used to characterize the correlation strength between different spatial regions in a video frame and the gaze point features and spatiotemporal features. The higher the value, the more likely the region is to be gazed upon. For example, if the spatiotemporal feature is [0.6, 0.4] and the weighted feature is [0.4, 0.24], the spatial attention matrix is ​​[[0.6×0.4, 0.6×0.24], [0.4×0.4, 0.4×0.24]] = [[0.24, 0.144], [0.16, 0.096]].

[0119] Step S2074: Determine the gaze probability distribution based on the spatial attention matrix and spatiotemporal features.

[0120] In some embodiments, the server determines the gaze probability distribution based on the spatial attention matrix and spatiotemporal features, which can be achieved in the following way: First, the spatial attention matrix and spatiotemporal features are subjected to a second weighted fusion to obtain fused features; then, the fused features are upsampled to obtain upsampled features; finally, the upsampled features are feature mapped to obtain the gaze probability distribution.

[0121] In this embodiment, the second weighted fusion (s) refers to the process of fusing the spatial attention matrix and spatiotemporal features according to preset weights, which are dynamically adjusted based on their importance to gaze prediction. The fused feature is the result of the second weighted fusion, simultaneously containing region association information from the spatial attention matrix and video content information from the spatiotemporal features; it is a more comprehensive intermediate feature. Upsampling is the process of upsampling the low-resolution fused features to the same resolution as the video frame using interpolation or mapping methods. The purpose of upsampling is to ensure that the fused features accurately correspond to each pixel or region of the video frame. The upsampled feature is the high-resolution feature obtained after upsampling, which can be directly mapped to each region of the video frame, preserving the core information of the fused features while matching the resolution to the video. Feature mapping is the process of converting the upsampled features into probability values ​​in the 0-1 range, which can be achieved through activation functions or normalization methods, ensuring that the output meets the basic requirements of a probability distribution.

[0122] Here, through a second weighted fusion, the regional association information of the spatial attention matrix and the video content information of spatiotemporal features can be integrated, avoiding the limitations of a single feature. Upsampling processing upscales low-resolution features to the video frame resolution, ensuring that the gaze probability accurately corresponds to each region. Feature mapping, through standardization and activation functions, makes the output conform to the basic requirements of the probability distribution. These steps together improve the accuracy and spatial precision of the gaze probability distribution, enabling more accurate prediction of the user's gaze tendency towards each region in subsequent video frames, providing a reliable basis for subsequent operations such as subtitle layout and content obstacle avoidance.

[0123] Here, pupil diameter changes are transformed into attention weights through feature mapping, which can quantify the impact of physiological responses on fixation; weighted processing can highlight key fixation point features; outer product operation can fuse spatiotemporal information and fixation features to generate an attention matrix that reflects spatial correlation; finally, the fixation probability distribution obtained by combining the matrix and spatiotemporal features can accurately predict the user's fixation tendency for each region in subsequent video frames, providing a reliable basis for subsequent subtitle layout, etc.

[0124] Here, by encoding gaze point sequences to extract users' historical gaze patterns, combining pupil diameter change rate to reflect attention intensity changes, and fusing spatiotemporal features of the second video frame sequence to capture content dynamics, the synergistic effect of multi-dimensional features improves the predictive accuracy of gaze probability distribution. This method can more accurately predict the user's attention area in subsequent video frame sequences, providing a reliable basis for dynamic subtitle layout and obstacle avoidance, thereby reducing subtitle interference with the user's visual focus and improving the effectiveness of information delivery.

[0125] In some embodiments, see Figure 7 , Figure 7The server may perform the following steps S701 to S703 before the eye-tracking data of the first video frame sequence in the target video based on the first object and the second video frame sequence in the target video:

[0126] Step S701: Determine the pupil diameter change rate of the first object relative to the first video frame sequence.

[0127] In this embodiment of the application, the specific meaning and implementation of the step of determining the pupil diameter change rate of the first object in relation to the first video frame sequence can be found in step S205 above, and will not be repeated here.

[0128] Step S702: Determine the length of the sliding window based on the rate of change of pupil diameter.

[0129] In this embodiment, the sliding window length refers to the time length used to capture the second video frame sequence, which is dynamically adjusted by the pupil diameter change rate to reflect the time range of subsequent video frames to be analyzed. For example, when the pupil diameter change rate is 0, the sliding window length Tw = 0.3 * (1 + 0) = 0.3 seconds; when the change rate is 1 mm / s, Tw = 0.3 * (1 + 0.5 * 1) = 0.45 seconds. Extending the sliding window allows for the capture of more subsequent frames.

[0130] Step S703: Based on the sliding window length, video frames are extracted starting from the terminating video frame in the first video frame sequence of the target video to obtain the second video frame sequence.

[0131] In this embodiment, video frame extraction refers to the process of extracting consecutive video frames from the target video, starting from the terminating video frame of the first video frame sequence and following a sliding window length. The purpose of video frame extraction is to determine the range of the second video frame sequence. The second video frame sequence refers to the set of consecutive video frames obtained through video frame extraction, located after the first video frame sequence, with a duration equal to the sliding window length and temporally continuous with the first video frame sequence. For example, if the first video frame sequence consists of frames from 4.7 to 5.0 seconds and the sliding window length is 0.3 seconds, then the second video frame sequence consists of 9 frames from 5.0 to 5.3 seconds.

[0132] Here, the sliding window length is dynamically adjusted by the pupil diameter change rate, allowing the time range of the second video frame sequence to adapt to the attentional changes of the first object (e.g., extending the window when the pupil dilates rapidly to capture more subsequent content; shortening the window when the pupil constricts to focus on recent frames), avoiding information redundancy or insufficiency that might occur with a fixed window length. This dynamic cropping method makes the second sequence more closely match the user's real-time attentional state, providing a more accurate analytical basis for subsequent steps such as gaze probability prediction based on the second sequence, and improving the adaptability and accuracy of the overall process.

[0133] In step S208, the server encodes the subtitle text corresponding to the second video frame sequence to obtain the text encoding vector of the subtitle text.

[0134] In this embodiment, encoding processing refers to the process of converting subtitle text into a computer-recognizable numerical vector form, with the aim of quantifying the semantic information of the subtitle text. The text encoding vector is the output of the encoding processing, and it contains semantic and syntactic information of the subtitle text.

[0135] In some embodiments, encoding the subtitle text to obtain a text encoding vector can be achieved in the following way: First, select a suitable text encoding model (e.g., Word2Vec or BERT); then, input the subtitle text into the text encoding model for word segmentation to obtain the word segmentation result; finally, map the word segmentation result through the embedding layer of the text encoding model to output a text encoding vector containing semantic and syntactic information.

[0136] In step S209, the server performs semantic parsing on the text encoding vector to obtain the part-of-speech probability and sentiment intensity value of each word in the subtitle text.

[0137] In this embodiment, semantic parsing refers to the process of analyzing the text encoding vector to extract deep semantic information such as the grammatical attributes and sentiment of words in the subtitle text. Part-of-speech probability refers to the probability value of a word in the subtitle text belonging to a specific part of speech (e.g., noun, verb, or adjective), reflecting the likelihood of the word being a certain part of speech. Sentiment intensity value is used to measure the strength of the emotion (e.g., positive, negative, or neutral) carried by words in the subtitle text, usually represented by a value between 0 and 1. A higher value indicates a higher sentiment intensity. For example, the positive sentiment intensity value for "joy" is 0.85, the negative sentiment intensity value for "disgust" is 0.92, and the neutral sentiment intensity value for "table" is 0.

[0138] In some embodiments, semantic parsing of text encoding vectors to obtain part-of-speech probability and sentiment intensity values ​​can be achieved as follows: First, the obtained text encoding vectors are input into a pre-trained semantic parsing model, which includes part-of-speech tagging and sentiment analysis modules; second, the text encoding vectors are processed layer by layer through the semantic parsing model, calculating the probability of each word belonging to different parts of speech; simultaneously, the sentiment tendency of each word is analyzed through the semantic parsing model, and the corresponding sentiment intensity value is output; finally, the part-of-speech probability and sentiment intensity value of each word are output.

[0139] In step S210, the server determines the subtitle priority of the subtitle text based on part-of-speech probability and sentiment intensity value.

[0140] In this embodiment, the subtitle priority is determined based on the part-of-speech and emotional information of the words in the subtitle text. The subtitle priority can be used to determine the display order, duration, or prominence of the subtitle text. For example, in news videos, subtitle text related to earthquake early warning has a higher subtitle priority than subtitle text related to weather because it contains key nouns and is related to an urgent emotion.

[0141] In some embodiments, determining the subtitle priority of subtitle text based on part-of-speech probability and sentiment intensity value can be achieved as follows: First, determine the part-of-speech of each word in the subtitle text based on part-of-speech probability, and determine the sentiment attribute of each word in the subtitle text based on sentiment intensity value; then, obtain the first weight value of part-of-speech and the second weight value of sentiment attribute; finally, use the first weight value and the second weight value to perform a weighted sum of part-of-speech probability and sentiment intensity to obtain the subtitle priority.

[0142] In this embodiment, part-of-speech refers to the category attribute of a word in the grammatical structure, reflecting the grammatical function of the word. For example, a part-of-speech can be a noun representing a person or thing, a verb representing an action, or an adjective representing a state. Emotional attribute refers to the emotional tendency carried by a word, usually divided into three categories: positive, negative, and neutral. Emotional attribute reflects the emotional coloring of a word. The first weight value is a pre-set numerical value used to measure the degree of influence of part-of-speech on subtitle priority; different parts of speech correspond to different first weight values. The second weight value is a pre-set numerical value used to measure the degree of influence of emotional attribute on subtitle priority; different emotional attributes correspond to different second weight values. For example, the second weight value for positive emotional words is 0.5, and the second weight value for neutral words is 0.1. Weighted summation refers to the calculation method of adding the product of the part-of-speech probability and the first weight value, and the product of the emotional intensity value and the second weight value, to obtain a comprehensive score. The subtitle priority is the result obtained through weighted summation, used to characterize the comprehensive score (0-1 range) of the importance of the subtitle text; a higher subtitle priority indicates that the subtitle text needs to be highlighted more.

[0143] In some embodiments, firstly, the part-of-speech probability of each word can be analyzed, and the part-of-speech with the highest probability can be selected as the final part-of-speech of the word; a threshold is set for the sentiment intensity value of each word, and the sentiment attribute is determined based on the threshold. Then, a part-of-speech weight table and a sentiment attribute weight table are preset based on the subtitle application scenario; according to the part-of-speech and sentiment attribute of each word, the corresponding first weight value and second weight value are matched from the weight table. Finally, the subtitle priority is obtained by weighted summation. For each word in the subtitle text, the product between the part-of-speech probability and the first weight value, and the product between the sentiment intensity value and the second weight value are calculated; the two products of each word are summed to obtain the comprehensive score of the word; the average of the comprehensive scores of all words is calculated and normalized to the 0-1 interval to obtain the subtitle priority of the subtitle text. For example, if the comprehensive scores of three words in the subtitle text are 0.9, 0.8, and 0.7, respectively, then the average is 0.8, and the subtitle priority is 0.8.

[0144] Here, by combining part-of-speech and sentiment attributes, and differentiating their contributions through weighting, the resulting subtitle priority more comprehensively reflects the importance of the subtitle text. This approach avoids the one-sidedness of a single-dimensional evaluation, making the determination of subtitle priority more accurate, providing a reasonable basis for the subsequent presentation of subtitles, and helping to improve the efficiency of information transmission.

[0145] Here, by encoding and semantically parsing the subtitle text, and combining part-of-speech probability and sentiment intensity values ​​to determine the subtitle priority, the importance of the subtitles can be accurately reflected. This helps to rationally arrange the display method of the subtitles and improve the effectiveness and relevance of the information conveyed by the subtitles.

[0146] In step S211, the server sequentially performs target detection on each video frame in the second video frame sequence to obtain the bounding box of each second object in the video frame and the semantic label of the bounding box.

[0147] In this embodiment, object detection is the process of identifying a second object in each video frame of a second video frame sequence and determining the location of the second object. A bounding box is a tool used to mark the location of the second object in each video frame using rectangular boxes, typically defined by the coordinates (x1, y1, x2, y2) of its top-left and bottom-right corners. A semantic label is a category name assigned to the detected second object; the semantic label is used to describe the semantic information of the second object.

[0148] In some embodiments, object detection is performed sequentially on each video frame in the second video frame sequence to obtain bounding boxes and semantic labels. This can be achieved as follows: First, the second video frame sequence is loaded, and each video frame is extracted in sequence; a pre-trained object detection model (e.g., YOLO or Faster R-CNN) is called to process the single-frame image. Through feature extraction and classification regression of the object detection model, the bounding box coordinates of all second objects in the video frame are output, and a corresponding semantic label is assigned to each bounding box. Finally, the bounding boxes and semantic labels of all second objects in the frame are obtained.

[0149] In step S212, the server determines the segmentation threshold corresponding to the bounding box based on the semantic labels.

[0150] In this embodiment, the segmentation threshold is a critical value used to distinguish the second object from the background during image segmentation. Pixel values ​​that meet the threshold condition are classified as the second object region, while those that do not are classified as the background. For example, for a second object with the semantic label of a face, the segmentation threshold can be set to 0.6, that is, the portion with a pixel value ≥ 0.6 is considered as the face region.

[0151] In some embodiments, determining the segmentation threshold corresponding to the bounding box based on semantic labels can be achieved in the following way: a mapping table between semantic labels and segmentation thresholds is preset. For the bounding box of each second object in each video frame, the semantic label of the bounding box is extracted, the value corresponding to the semantic label is queried from the mapping table, and the value is determined as the segmentation threshold of the bounding box.

[0152] In other embodiments, determining the segmentation threshold corresponding to the bounding box based on semantic labels can also be implemented in the following way: First, a mapping table between semantic labels and features to be extracted is preset. For different semantic labels, different features to be extracted can be retrieved from this mapping table. Then, a segmentation threshold is automatically determined based on the features to be extracted using a preset adaptive threshold segmentation algorithm. For example, for a broadcaster's face, a suitable segmentation threshold is automatically determined based on the grayscale value or color features of the face using an adaptive threshold segmentation algorithm. Based on this segmentation threshold, the face can be segmented from the background to obtain the accurate outline of the face. For text areas on a large screen, a segmentation threshold is determined based on features such as the contrast between the text and the background using an adaptive threshold segmentation algorithm. Based on this segmentation threshold, the text can be segmented from the background to obtain the accurate outline of the text area. The specific limitations of the adaptive threshold segmentation algorithm are not specified here.

[0153] In step S213, the server performs image segmentation on the image region corresponding to the bounding box based on the segmentation threshold to obtain the contour region of each second object in the video frame.

[0154] In this embodiment, image segmentation is the process of dividing the image region corresponding to the bounding box into different pixel groups, with each group of pixels corresponding to an independent second object or background. The contour region is the complete pixel region enclosed by the outermost boundary of the second object, which fits the target shape more precisely than the bounding box. For example, segmenting the bounding box region containing an apple separates the apple pixels from the background pixels, resulting in a pixel region containing only the apple.

[0155] In some embodiments, image segmentation is performed on the image region corresponding to the bounding box based on a segmentation threshold to obtain a contour region. This can be achieved in the following way: First, determine the pixel values ​​in the image region corresponding to the bounding box; then, compare each pixel value with the segmentation threshold and retain pixels that meet the threshold conditions; finally, perform connectivity analysis on the retained pixels, merge adjacent target pixels to form a complete closed region, which is the contour region of the second object in the video frame.

[0156] Here, through the above steps, the segmentation threshold can be set specifically by combining the semantic information of the second object, thereby accurately extracting the contour region of the second object in the video frame and improving the accuracy and adaptability of target contour recognition.

[0157] In step S214, the server determines the set of candidate subtitle parameters for each video frame in the second video frame sequence.

[0158] In this embodiment, the candidate subtitle parameter set refers to a series of possible parameter combinations generated for the subtitle text in the video frame. Each parameter combination includes key attributes such as the position, font size, and transparency of the subtitle, which are used to subsequently select the optimal subtitle parameters.

[0159] In some embodiments, the candidate caption parameter set can be determined as follows: First, 1000 initial candidate solutions pre-generated by NSGA-II during the offline phase are used as a basic experience pool, covering caption display methods in different scenarios. When processing the second video frame sequence online, NSGA-II local evolution is triggered every 5 frames to sort the candidate solutions in the basic experience pool according to their quality and calculate their distribution density, leaving the best candidate solution. Then, new candidate solutions are generated through filtering, combination, and fine-tuning to supplement the basic experience pool, ensuring the diversity of solutions to adapt to changes in the picture. Finally, for the second video frame sequence to be processed, suitable candidate solutions are selected from the experience pool based on features such as the position of prominent objects in the picture and the area that the user may be looking at, ultimately forming a candidate caption parameter set containing dozens of sets of parameters.

[0160] In step S215, the server determines the subtitle parameters of the subtitle text in the video frame from the candidate subtitle parameter set based on a preset objective function.

[0161] In this embodiment, the objective function is a mathematical function used to evaluate the merits of candidate subtitle parameters. The objective function includes: a first penalty term negatively correlated with the occlusion area, a reward term positively correlated with subtitle readability, and a second penalty term negatively correlated with the difference. The occlusion area is determined based on the contour region, subtitle readability is determined based on the gaze probability distribution and subtitle priority, and the difference is determined based on the subtitle parameters of the video frame and the adjacent video frames preceding it. The occlusion area refers to the overlap area between the subtitle region and the contour region of the second object in the video frame, reflecting the degree to which the subtitle occludes key content.

[0162] The first penalty term, negatively correlated with the occlusion area in the objective function, is used to suppress caption parameters that obscure key objects. Caption readability refers to the degree to which the caption text is clearly identifiable by the user, influenced by caption position, font size, and caption priority. The reward term, positively correlated with caption readability in the objective function, encourages easily readable caption parameters. Difference refers to the degree of difference between the caption parameters of the current video frame and those of the previous video frame (e.g., positional offset or size change), reflecting the stability of the caption text. The second penalty term, negatively correlated with difference, is used to suppress abrupt changes in caption parameters, ensuring inter-frame coherence.

[0163] In some embodiments, see Figure 8 , Figure 8 The following step S215 illustrates how the server determines the subtitle parameters of the subtitle text in the video frame from the candidate subtitle parameter set based on a preset objective function. This can be achieved by executing the following steps S2151 to S2152:

[0164] Step S2151: For each candidate subtitle parameter in the candidate subtitle parameter set, determine the target value of the candidate subtitle parameter under the objective function.

[0165] In this embodiment, candidate subtitle parameters refer to a specific set of subtitle attributes in the candidate subtitle parameter set, including subtitle position, font size, and transparency, etc. Candidate subtitle parameters are also possible subtitle setting schemes that may be selected. The target value refers to the calculation result obtained by substituting the candidate subtitle parameters into the target function, which is used to measure the quality of the candidate subtitle parameters. The higher the target value, the more the candidate subtitle parameters meet the requirements.

[0166] In some embodiments, see Figure 9 , Figure 9 The example shown in step S2151 illustrates that the server determines the target value of the candidate caption parameters under the objective function, which can be achieved by executing the following steps S901 to S906:

[0167] Step S901: Based on the candidate subtitle parameters, determine the candidate subtitle size, candidate transparency, candidate subtitle area, and candidate center point coordinates of the subtitle text in the video frame.

[0168] In this embodiment, the candidate subtitle size refers to the font size of the subtitle as defined in the candidate subtitle parameters, typically in pixels. The candidate subtitle size determines the visual size of the subtitle and directly affects its readability. Candidate transparency refers to the degree of transparency of the subtitle as defined in the candidate subtitle parameters, which can be used to balance the blending of the subtitle text with the background. The candidate subtitle region refers to the rectangular area occupied by the candidate subtitle text in the video frame, defined by the coordinates of the upper left and lower right corners, reflecting the spatial extent of the subtitle text. The candidate center point coordinates refer to the geometric center coordinates of the candidate subtitle region, used to measure the positional stability of the subtitle.

[0169] In some embodiments, the candidate subtitle size, candidate transparency, candidate subtitle region, and candidate center point coordinates of the subtitle text in the video frame are determined based on the candidate subtitle parameters. This can be achieved by: parsing the candidate subtitle parameters and extracting the font size defined therein as the candidate subtitle size; extracting the transparency value in the parameters as the candidate transparency; calculating the candidate subtitle region based on the position information in the candidate subtitle parameters; and calculating the center point using the coordinates of the candidate subtitle region to obtain the coordinates of the candidate center point.

[0170] Step S902: Based on the gaze probability distribution, determine the visual focus coordinates of the first object in the video frame.

[0171] In this embodiment, the visual focus coordinates are the pixel coordinates of the first object most likely to be gazed at, determined based on the gaze probability distribution. The visual focus is the point in the video frame where attention is most concentrated.

[0172] In some embodiments, the visual focus coordinates of the first object in the video frame are determined based on the gaze probability distribution. This can be achieved by: first, traversing the gaze probability distribution of the video frame and recording the gaze probability value of each region; then, filtering out the region with the highest gaze probability value, and if the region is a grid, taking the center point of the grid as the visual focus coordinates.

[0173] Step S903: Determine the first penalty value based on the candidate subtitle region, the outline region, and the first penalty item.

[0174] In this embodiment, the first penalty value is a penalty value calculated based on the occlusion area between the candidate subtitle region and the second object contour region, used to suppress the occlusion of subtitle parameters of key objects. The larger the occlusion area, the smaller the first penalty value.

[0175] In some embodiments, the first penalty value is determined based on the candidate caption region, the contour region, and the first penalty term. This can be achieved by: first, calculating the intersection-union ratio (IUU) of the candidate caption region with the contour regions of all second objects; then, substituting the IUU into the first penalty term to obtain the first penalty value.

[0176] Step S904: Determine the reward value based on candidate caption size, candidate transparency, caption priority, coordinate distance between visual focus coordinates and candidate center point coordinates, and reward items.

[0177] In this embodiment, the reward value is a calculated value that takes into account the size, transparency, priority, and distance from the visual focus of the candidate subtitles, and is used to encourage easily readable subtitle parameters. The higher the readability of the subtitle, the greater the reward value.

[0178] In some embodiments, the reward value can be determined by: first, calculating the coordinate distance between the visual focus coordinates and the candidate center point coordinates; then, substituting the candidate caption size, candidate transparency, caption priority, and the coordinate distance into the reward item to obtain the reward value.

[0179] Step S905: Determine the second penalty value based on the candidate center point coordinates, the center point coordinates of the subtitle text in adjacent video frames, and the second penalty term.

[0180] In this embodiment, the second penalty value is a penalty value calculated based on the difference between the coordinates of the candidate subtitle center point in the current video frame and the coordinates of the subtitle center point in the previous video frame, used to suppress abrupt changes in subtitle position. The greater the difference, the smaller the second penalty value.

[0181] In some embodiments, the second penalty value can be determined by substituting the candidate center point coordinates and the center point coordinates of the subtitle text in adjacent video frames into the second penalty term to obtain the second penalty value.

[0182] Step S906: Determine the target value based on the first penalty value, the reward value, and the second penalty value.

[0183] In this embodiment, the target value is a comprehensive value determined based on a first penalty value, a reward value, and a second penalty value, used to measure the overall quality of the candidate subtitle parameters. The higher the target value, the better the candidate subtitle parameters.

[0184] In some embodiments, the target value is determined based on the first penalty value, the reward value, and the second penalty value, which can be achieved by: first, obtaining the third weight value of the first penalty value, the fourth weight value of the reward value, and the fifth weight value of the second penalty value; then, using the third weight value, the fourth weight value, and the fifth weight value, performing a weighted summation on the first penalty value, the reward value, and the second penalty value to obtain the target value.

[0185] In this embodiment, the third weight value is a preset value used to measure the influence of the first penalty value on the target value; the more the occlusion problem needs to be avoided, the higher the third weight value. The fourth weight value is a preset value used to measure the influence of the reward value on the target value; the more important the subtitle readability, the higher the fourth weight value. The fifth weight value is a preset value used to measure the influence of the second penalty value on the target value; the higher the subtitle stability requirement, the higher the fifth weight value. Weighted summation refers to the calculation method of multiplying the first penalty value, reward value, and second penalty value by their corresponding third, fourth, and fifth weight values, respectively, and then summing them to obtain the total. The target value is the result of weighted summation, and the target value comprehensively reflects the overall performance of the candidate subtitle parameters in terms of occlusion control, subtitle readability, and positional stability.

[0186] In some embodiments, firstly, a weight table can be preset according to the video scene type, and key features of the current video frame can be detected. The weight table is then fine-tuned based on the key features. Corresponding values ​​are extracted from the adjusted weight table as the final third, fourth, and fifth weight values. Then, the product of the first penalty value and the third weight value, the product of the reward value and the fourth weight value, and the product of the second penalty value and the fifth weight value are calculated respectively. Finally, the three product results are added together, and the sum is the target value.

[0187] Here, by assigning weights to the first penalty value, reward value, and second penalty value and then summing them, the priorities of occlusion control, subtitle readability, and positional stability can be dynamically adjusted according to the video scene, making the target values ​​more aligned with actual needs. This approach avoids the one-sidedness of a single-dimensional evaluation, ensuring that the final subtitle parameters achieve a balance in multiple aspects and improving the overall effect of subtitle presentation.

[0188] Here, by decomposing the key attributes of candidate caption parameters and combining the gaze probability distribution, contour region, and adjacent frame parameters, a first penalty value, a reward value, and a second penalty value are calculated respectively. Finally, a target value is obtained by combining these factors, which can comprehensively measure the quality of candidate parameters. This method ensures that the determined caption parameters reduce the obstruction of key content, conform to user visual habits to improve readability, and maintain inter-frame stability, thereby optimizing the caption presentation effect and improving user experience and information delivery efficiency.

[0189] Step S2152: The candidate subtitle parameters with the largest target value are determined as the subtitle parameters of the subtitle text in the video frame.

[0190] In this embodiment, the maximum target value is the largest among all candidate subtitle parameter target values, and the candidate subtitle parameter corresponding to the maximum target value is the current optimal selection. The subtitle parameters are the parameters ultimately determined from the candidate set and used for rendering the subtitles.

[0191] In some embodiments, the subtitle parameters can be determined in the following ways: First, traverse each set of candidate subtitle parameters in the candidate subtitle parameter set, substitute each candidate subtitle parameter into a preset objective function, and calculate the corresponding objective value; then, compare the objective values ​​of all candidate subtitle parameters, select the objective value with the largest value, and record it as the maximum objective value; finally, obtain the set of candidate subtitle parameters corresponding to the maximum objective value, and determine it as the final subtitle parameter of the subtitle text in the video frame.

[0192] Here, by calculating the target value of each candidate subtitle parameter and selecting the parameter corresponding to the maximum target value, it can be ensured that the final subtitle parameters are optimal in terms of occlusion control, subtitle readability, and coherence. This reduces the occlusion of key video content, ensures that users can clearly obtain information, maintains the stability of subtitle display, and improves the overall viewing experience.

[0193] Here, by generating a set of candidate subtitle parameters and filtering them based on an objective function, and taking into account factors such as occlusion area, subtitle readability, and inter-frame differences, the optimal subtitle parameters that balance obstacle avoidance, readability, and coherence can be determined, thereby improving the user's viewing experience and the effectiveness of information delivery.

[0194] In step S216, the server renders subtitles on the second video frame sequence based on the subtitle parameters to obtain the subtitle rendering result.

[0195] In this embodiment of the application, the specific meaning and implementation of the step of rendering subtitles on the second video frame sequence based on subtitle parameters can be found in step S105 above, and will not be repeated here.

[0196] Step S217: The server sends the subtitle rendering result to the terminal.

[0197] Step S218: The terminal displays the subtitle rendering result.

[0198] Here, by combining the gaze probability distribution, the outline region of the second object, and the subtitle priority, the optimal subtitle parameters that balance obstacle avoidance, readability, and coherence are selected from the candidate subtitle parameter set based on the objective function. This makes the subtitle layout conform to the user's visual habits, dynamically avoid key content, and highlight important information, while ensuring the coherence and naturalness of the presentation. Ultimately, this effectively improves the smoothness of video viewing, the effectiveness of information delivery, and the completeness of content, bringing users a better video interaction experience.

[0199] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0200] This application provides an intelligent real-time obstacle avoidance method for subtitles based on multimodal perception and dynamic collaborative optimization. To achieve accurate presentation and dynamic optimization of video subtitles, this method constructs a full-chain intelligent solution from content understanding to visual presentation. First, it captures user attention distribution and scene semantics through intelligent perception, that is, it adopts spatiotemporal feature modeling, divides 5 frames of video into 16×16×5 spatiotemporal blocks with three-dimensional cubic convolution, extracts spatial texture features with 3×3×1 convolution and extracts temporal motion features with 1×1×3 convolution, and after dynamic weighted fusion, it improves the dynamic scene focus capture capability compared with traditional two-dimensional convolution. The focus recognition accuracy in motion-blurred scenes is better than the traditional average fusion method. At the same time, a dynamic time window mechanism is introduced, which expands the basic window of 0.3 seconds to 0.15-0.45 seconds according to the pupil diameter change rate. In the case of rapid attention switching, the response speed is better than the traditional fixed window scheme (the latency can be controlled within 100ms). Combining video features and eye movement vector outer product to generate a spatial attention matrix, the heat map resolution is improved to 1 / 4 of the original image, which significantly improves the positioning accuracy of key areas. Secondly, a multi-objective optimization approach is adopted to dynamically balance occlusion, readability, and stability through intelligent decision-making. Specifically, a cascaded Mask R-CNN is used to simultaneously detect faces, text regions, and moving targets in video frames to improve detection accuracy. A bidirectional encoder representations from transformers-conditional random field (BERT-CRF) hybrid model is used for contextual semantic encoding and part-of-speech tagging, improving the accuracy of sentiment intensity recognition and reducing information density evaluation errors. Furthermore, a collaborative mechanism integrates global search Pareto optimal solution sets with real-time policy adjustments to reduce occlusion area, improve subtitle readability and positional stability, and ensure subtitles are located within a safe area of ​​the screen through state-action space design, while a stepped curve reduces visual interference. Finally, GPU acceleration enables accurate rendering and dynamic obstacle avoidance, significantly improving video content production and interactive experience.

[0201] First, multimodal input information is collected through a multimodal data acquisition layer. The input sources include video streams containing RGB and depth information, audio streams, and original unaligned subtitle text. On the hardware side, spatial depth perception is achieved through binocular cameras, sound source localization is completed through microphone arrays, and user head posture is detected by IMU sensors, providing basic data for subsequent processing.

[0202] Then, visual focus detection is performed through a visual focus detection module, based on an improved Spatio-Temporal Convolution Attention Network (ST-CAN). The input data for this network consists of a video frame sequence and user eye-tracking data estimated by an eye tracker or camera. The output data is a heatmap of the current video frame's visual hotspot probability distribution. The specific execution logic is as follows: First, input data preprocessing and alignment are performed. The video frame sequence is sampled at 30fps with a resolution of 1920×1080, using a dual-channel input of RGB + depth map. User eye-tracking data is obtained through an eye tracker or a regular camera (based on a convolutional neural network-based gaze estimation algorithm). The user eye-tracking data includes the current gaze coordinates (x, y) and the pupil diameter used for attention intensity estimation. Next, time synchronization is performed, using hardware timestamps to align the video frames with the eye-tracking data with an error <10ms. Spatial mapping is then completed, converting the screen coordinate system of the eye-tracking device to the pixel coordinate system of the video frame.

[0203] Next, in the spatiotemporal feature extraction stage, when processing video frames, five consecutive video frames (approximately 0.17 seconds in length) are divided into overlapping three-dimensional cubes (16×16 pixels × 5 frames). Branch 1 uses a 3×3×1 convolutional kernel to extract local texture features of a single frame, i.e., spatial features; Branch 2 uses a 1×1×3 convolutional kernel to extract cross-frame motion change features, i.e., temporal features. Finally, spatial and temporal features are dynamically weighted and fused through channel attention gating to obtain the video feature map. When processing eye-tracking data, the gaze point sequence (30 points) within the past second is input into a Long Short-Term Memory (LSTM) network, outputting a 128-dimensional latent vector. Then, through attention intensity encoding, the pupil diameter change rate is mapped to 0-1 attention weights using the Sigmoid function, and these attention weights are used to weight the latent vectors to obtain weighted features.

[0204] Subsequently, spatiotemporal attention calculation is performed. In the joint attention layer, the video feature map and weighted features (128 dimensions) are outer-producted to generate a spatial attention matrix. The spatial attention matrix and the video feature map are then weighted and fused to obtain a low-resolution feature map. Finally, a deconvolution layer upsamples the low-resolution feature map to the original image size, outputting the gaze probability of each pixel, thus completing the probability distribution prediction and generating a visual hotspot probability distribution map. Additionally, based on the pupil diameter change rate, according to formula T... w =T b *(1+0.5*pupil diameter change rate), adjust the sliding window length T in real time. w Among them, T bThe base window length is typically T. b =0.3 seconds, the length T of the sliding window can be adjusted. w Dynamically expands to 0.15-0.45 seconds. Furthermore, based on the sliding window length T... w Extract consecutive video frames from the video for subsequent processing.

[0205] Next, the scene semantic understanding module performs image salient object detection and caption semantic keyness evaluation. The specific implementation process is as follows: For image salient object detection, a cascaded Mask R-CNN algorithm combined with adaptive threshold segmentation is adopted, and the output is the contours of salient objects in the video frame and semantic labels (e.g., faces, text regions, and moving objects). First, the input video frame is acquired. Taking a news report video as an example, its frame may contain the anchor's face, text regions on the large screen behind him, and moving objects in the on-site interview.

[0206] Next, a cascaded Mask R-CNN model is used to perform preliminary object detection on the video frames, identifying salient objects in the scene, such as the anchor's face, text regions on the large screen, and people in the on-site interview. Bounding boxes are generated for each detected salient object; for example, a bounding box exactly enclosing the anchor's face is generated, and text regions on the large screen are identified as a whole and bounding boxes are generated accordingly. Then, an adaptive thresholding segmentation algorithm is used to further process each detected salient object to obtain precise contours. For example, for the anchor's face, an appropriate threshold is automatically determined based on its grayscale value or color features to segment the face from the background and obtain a precise contour; for the text regions on the large screen, a threshold is determined by analyzing features such as the contrast between the text and the background to obtain accurate contours of the text regions. Finally, the precise contours and semantic labels of salient objects in the video frames are output, such as the anchor's face contour and face label, the text region contour and text region label on the large screen, and the contours and moving object labels of the crowd in the on-site interview.

[0207] In terms of subtitle semantic criticality evaluation, a BERT-CRF hybrid model is adopted to analyze the sentiment intensity and information density of the subtitle text, outputting a dynamic priority score (0-1 range), i.e., subtitle priority. First, the input subtitle text is obtained, such as a subtitle from a news report video: "Today we will introduce a new artificial intelligence algorithm called Transformer, which has achieved great success in the field of natural language processing." Then, the subtitle text is semantically encoded using the BERT model, converting it into a vector representation to capture semantic information. For example, "Transformer" is a proper noun representing a specific artificial intelligence algorithm. Next, based on the BERT encoding, the CRF model considers the contextual information of the subtitle text and labels the part-of-speech and sentiment of each word. For example, the CRF model identifies "great success" as expressing a positive sentiment. Simultaneously, the CRF model increases the weight of proper nouns according to preset rules, because "Transformer" is core content with high information density. Finally, after model analysis, the dynamic priority score of the subtitle text is output. Assuming the score of this paragraph is 0.8, it indicates that the subtitle text contains important scientific and technological information and has high semantic criticality.

[0208] Furthermore, this application embodiment introduces a dynamic obstacle avoidance decision engine. The optimization objective function of this dynamic obstacle avoidance decision engine is R = α × Min (occlusion area) + β × Max (subtitle readability) + γ × Min (position jump frequency), where α, β, and γ are weight coefficients used to balance different optimization objectives. The solution algorithm adopts a combination of multi-objective genetic algorithm and real-time reinforcement learning. For example, the multi-objective genetic algorithm can be Non-dominated Sorting Genetic Algorithm II (NSGA-II), and the real-time reinforcement learning can be Proximal Policy Optimization (PPO). The state space covers the visual hotspot distribution, salient object positions, and subtitle priority, while the action space includes subtitle coordinates (x, y), font size, and transparency gradient curve.

[0209] The mathematical modeling process for optimizing the objective function is described in detail below. The formula for minimizing the occlusion area is as follows: (1)

[0210]

[0211] Where S represents the occlusion area, and IoU represents the intersection-over-union ratio, used to measure the subtitle bounding box B. text The contour region B of the i-th salient object objectiThe degree of overlap; N is the total number of salient objects in the video frame; ω i These are weight parameters that are dynamically adjusted based on the importance of the i-th salient object. For example, the weight of a face is ω = 0.8, and the weight of text is ω = 0.5. MASK(B objecti (∈Saliency Object) represents a mask function, when B objecti When an object is a prominent object, the masking function will mark or mask it to distinguish the areas in the image that need to be protected and prevent them from being obscured by elements such as subtitles or text.

[0212] The formula for maximizing subtitle readability is: Subtitle Readability = Font Size * (1 - Transparency) * Semantic Keyness / Distance Coefficient, where the distance coefficient... x text and y text The coordinates of the subtitle text, x focus and y focus These are the coordinates of the visual focus. The closer the subtitle text is to the visual focus, the smaller the distance coefficient, indicating higher readability. Semantic criticality is a dynamic priority score in the 0-1 range calculated using the BERT model described above.

[0213] The formula for minimizing the position jump frequency is as follows (2):

[0214]

[0215] Where f represents the position transition frequency, x t and y t It is the coordinate of the current frame's subtitle text, x t-1 and y t-1 Here, is the coordinate of the subtitle text in the previous frame, and T is the statistical duration. The constraint on the position jump frequency is: when the position change between adjacent frames exceeds 10% of the screen width, a penalty mechanism is triggered. Simultaneously, an exponential smoothing mechanism is introduced to reduce high-frequency jitter. The formula for this exponential smoothing mechanism is expressed as follows: in, x represents the smoothed coordinates of the subtitle text in the current frame. t-1 x represents the coordinates of the subtitle text in the previous frame. t The optimal position coordinates of the subtitle text in the current frame are calculated.

[0216] The multi-objective optimization algorithm collaborative mechanism in this embodiment achieves global optimization and real-time dynamic adjustment of subtitle display through deep collaboration between the NSGA-II multi-objective genetic algorithm and PPO real-time reinforcement learning. The specific workflow is as follows:

[0217] During the offline initialization phase, NSGA-II pre-generates 1000 initial candidate solutions based on a preset optimization objective function using a non-dominated sorting genetic algorithm. These candidate solutions cover subtitle display strategies under different scenarios, forming an initial experience pool. At the same time, using a supervised learning method, the Pareto optimal solution set generated by NSGA-II is used as training data to initialize the policy network of PPO, enabling it to have basic policy selection capabilities.

[0218] After entering the online optimization phase, NSGA-II local evolution is triggered every 5 frames of video data processed. Through non-dominated sorting and crowding calculation, the current optimal solution set is preserved, and offspring populations are generated through selection, crossover, and mutation operations. This process can maintain population diversity to adapt to dynamic changes in the scene. During the processing of each frame of video, PPO selects actions (i.e., the above-mentioned visual hotspot distribution, salient object positions, and caption priority) based on the current state (i.e., the above-mentioned caption position, font size, and transparency gradient curve), and updates the policy network parameters by interacting with the environment (i.e., calculating reward values) to improve real-time decision-making capabilities.

[0219] The objective function is dynamically adjusted using a multi-objective fusion reward, specifically formulated as R = α × (1 - occlusion area) + β × subtitle readability + γ × (1 - jump frequency). Here, α, β, and γ are weight coefficients dynamically adjusted based on scene classification; for example, α = 0.6 for interview videos and β = 0.7 for data-driven videos. Furthermore, a fuzzy logic system is used to handle weight conflicts; for instance, when the occlusion area > 20%, the α weight can be automatically increased by 50%.

[0220] This collaborative optimization has several advantages. In terms of balancing global and local factors, NSGA-II is responsible for globally searching for the Pareto optimal solution set, ensuring long-term policy optimization, while PPO adjusts its actions through real-time feedback to cope with short-term scenario changes. In terms of data-driven continuous evolution, NSGA-II's candidate solution set provides initial experience for PPO, and PPO's online learning results are fed back to NSGA-II, forming a closed-loop optimization. At the same time, by dynamically adjusting the objective function weights and policy network parameters, the system has the ability to adapt to different scenarios, such as education, live streaming, or news.

[0221] Table 1. State Space and Action Space Design

[0222] Dimension Representation Range of values Visual hotspot distribution Gaussian mixture model parameters (μ, Σ) Spatial coordinates range from 0 to 1 Prominent object location Bounding box coordinates (x, y, w, h) Screen ratio value is 0-1 Subtitle priority Semantic Keyness Score floating-point value Historical action trajectory Position coordinate sequence of the first 3 frames Spatial coordinates range from 0 to 1

[0223] The dynamic obstacle avoidance decision engine constructs a full-chain intelligent subtitle presentation solution around aspects such as state space and action space design. Regarding the state space and action space design, as shown in Table 1, the state space dimension encompasses visual hotspot distribution (represented by Gaussian mixture model parameters (μ, Σ), with spatial coordinate values ​​ranging from 0 to 1), salient object positions (represented by bounding box coordinates (x, y, w, h), with screen ratio values ​​ranging from 0 to 1), subtitle priority (represented by semantic key score, a floating-point value), and historical action trajectory (presented as a sequence of position coordinates from the previous 5 frames, with spatial coordinate values ​​ranging from 0 to 1). Action space parameterization includes continuous and discrete actions. For continuous actions, the subtitle coordinates (x, y) are... Sampling is performed using a beta distribution (mean μ = 0.5, concentration parameter α = 2), with a font size dynamic range of 16-32px (step size 2px). For discrete actions, three types of transparency gradient curves are available: linear, sinusoidal, and stepwise, with a gradient duration of 0.5-3 seconds (5 discrete levels). Constraints are also set: a safe area constraint prohibits display within 20% of the screen edge and maintains a distance of at least 5% of the screen width from salient objects; a physiological constraint requires a font size ≥ 16px to ensure readability and transparency ≥ 30% to avoid complete transparency. Utilizing the NSGA-II+PPO collaborative mechanism, the global search Pareto optimal solution set (NSGA-II) is combined with real-time policy adjustment (PPO) to reduce occlusion area, improve readability and positional stability. State-action space design ensures the subtitles remain within the safe screen area, and the stepwise curve reduces visual interference.

[0224] Finally, the rendering output layer begins. First, the Vulkan API is used to build the GPU rendering pipeline and initialize hardware resources. Pre-allocate video memory to store subtitle textures and path data, and establish an asynchronous communication mechanism between the CPU and GPU. After initialization, the main thread receives subtitle parameters from the dynamic obstacle avoidance engine and object contour coordinates from the scene understanding module. It then starts the real-time rendering pipeline, converting the text into 3D Bézier curve contours and generating anti-aliased textures with depth information. During real-time rendering, B-spline fitting is performed based on the contour regions of salient objects. A safe distance (5% screen width + semantic weight correction) is calculated to generate a wraparound path. Frame-by-frame detection of occlusion between subtitles and salient objects is performed (path replanning is triggered when IoU > 0.1) to ensure real-time spatial obstacle avoidance.

[0225] Simultaneously, visual focus switching is dynamically detected using heatmap KL divergence, and switching time points are predicted based on historical data. According to subtitle priority, high-priority subtitles are preloaded 500ms before switching and inserted within 200ms after switching, using a rapid sine curve fade-in. Low-priority subtitles fade in linearly during non-switching periods. Based on spatial and temporal obstacle avoidance results, the rendering strategy is dynamically adjusted through multimodal data fusion (50% visual focus + 30% object position + 20% subtitle priority). When quality indicators (i.e., occlusion area < 5%, subtitle readability > 0.7, positional stability > 95%) continuously fail to meet standards, NSGA-II local evolution and PPO strategy updates are triggered for collaborative optimization. Ultimately, rendering latency is strictly controlled, obstacle avoidance accuracy is improved, and user testing has verified the achievement of efficient and accurate subtitle rendering output.

[0226] Overall, the embodiments of this application capture user attention distribution and scene semantics in real time through intelligent perception, and optimize multiple objectives such as occlusion area, subtitle readability and position stability through intelligent decision-making. Relying on GPU acceleration, it achieves accurate rendering and dynamic obstacle avoidance, forming a full-chain intelligent solution from content understanding to visual presentation, bringing revolutionary improvements to video content production and interactive experience.

[0227] It is understood that in the embodiments of this application, if the content involving user information, such as target video, gaze probability distribution, subtitle priority, outline region and subtitle parameters, involves data related to user information or enterprise information, when the embodiments of this application are applied to specific products or technologies, it is necessary to obtain user permission or consent, or to obfuscate this information to eliminate the correspondence between this information and the user; and the collection and processing of related data should strictly comply with the requirements of relevant national laws and regulations when applied in practice, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0228] The following description continues to illustrate the exemplary structure of the video processing apparatus 455 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2As shown, the software modules stored in the video processing device 455 in the memory 450 may include: a first determining module 4551, configured to determine the gaze probability distribution of the first object in each region of the second video frame sequence based on eye-tracking data of the first object for a first video frame sequence in the target video and a second video frame sequence in the target video; the second video frame sequence is an adjacent video frame sequence located after the first video frame sequence; a second determining module 4552, configured to determine the subtitle priority of the subtitle text corresponding to the second video frame sequence; the subtitle priority is used to characterize the importance of the subtitle text in the target video; a video frame recognition module 4553, configured to sequentially perform video frame recognition for each video frame in the second video frame sequence to obtain the contour region of each second object in the video frame; a third determining module 4554, configured to determine the subtitle parameters of the subtitle text in the second video frame sequence based on the gaze probability distribution, the subtitle priority, and the contour region; and a subtitle rendering module 4555, configured to render subtitles in the second video frame sequence based on the subtitle parameters.

[0229] In some embodiments, the eye-tracking data includes a fixation point sequence and a pupil diameter sequence; the first determining module 4551 is further configured to: encode the fixation point sequence to obtain the fixation point features of the first object in relation to the first video frame sequence; determine the pupil diameter change rate of the first object in relation to the first video frame sequence based on the pupil diameter sequence; extract spatiotemporal features from the second video frame sequence to obtain the spatiotemporal features of the second video frame sequence; and determine the fixation probability distribution based on the fixation point features, the pupil diameter change rate, and the spatiotemporal features.

[0230] In some embodiments, the first determining module 4551 is further configured to: perform spatiotemporal segmentation processing on the second video frame sequence to obtain spatiotemporal blocks corresponding to the second video frame sequence; extract temporal features from the spatiotemporal blocks to obtain temporal features of the second video frame sequence; the temporal features are used to characterize the dynamic change information between consecutive video frames in the second video frame sequence; extract spatial features from the spatiotemporal blocks to obtain spatial features of the second video frame sequence; the spatial features are used to characterize the static visual information in each video frame in the second video frame sequence; and perform a first weighted fusion on the temporal features and the spatial features to obtain the spatiotemporal features.

[0231] In some embodiments, the first determining module 4551 is further configured to: perform feature mapping on the pupil diameter change rate to obtain attention weights; perform weighted processing on the fixation point features based on the attention weights to obtain weighted features; perform an outer product operation on the spatiotemporal features and the weighted features to obtain a spatial attention matrix; and determine the fixation probability distribution based on the spatial attention matrix and the spatiotemporal features.

[0232] In some embodiments, the first determining module 4551 is further configured to: perform a second weighted fusion on the spatial attention matrix and the spatiotemporal features to obtain fused features; perform upsampling processing on the fused features to obtain upsampled features; and perform feature mapping on the upsampled features to obtain the gaze probability distribution.

[0233] In some embodiments, the apparatus 455 further includes a cropping module, the cropping module being configured to: determine the pupil diameter change rate of the first object in relation to the first video frame sequence; determine a sliding window length based on the pupil diameter change rate; and crop video frames starting from the terminating video frame of the first video frame sequence in the target video based on the sliding window length to obtain the second video frame sequence.

[0234] In some embodiments, the second determining module 4552 is further configured to: encode the subtitle text corresponding to the second video frame sequence to obtain a text encoding vector of the subtitle text; perform semantic parsing on the text encoding vector to obtain the part-of-speech probability and sentiment intensity value of each word in the subtitle text; and determine the subtitle priority of the subtitle text based on the part-of-speech probability and the sentiment intensity value.

[0235] In some embodiments, the second determining module 4552 is further configured to: determine the part-of-speech of each word in the subtitle text based on the part-of-speech probability, and determine the sentiment attribute of each word in the subtitle text based on the sentiment intensity value; obtain a first weight value of the part-of-speech and a second weight value of the sentiment attribute; and use the first weight value and the second weight value to perform a weighted summation of the part-of-speech probability and the sentiment intensity to obtain the subtitle priority.

[0236] In some embodiments, the video frame recognition module 4553 is further configured to: sequentially perform target detection on each video frame in the second video frame sequence to obtain the bounding box of each second object in the video frame and the semantic label of the bounding box; determine the segmentation threshold corresponding to the bounding box based on the semantic label; and perform image segmentation on the image region corresponding to the bounding box based on the segmentation threshold to obtain the contour region of each second object in the video frame.

[0237] In some embodiments, the third determining module 4554 is further configured to: for each video frame in the second video frame sequence, determine a set of candidate subtitle parameters for the subtitle text in the video frame; and, based on a preset objective function, determine the subtitle parameters of the subtitle text in the video frame from the set of candidate subtitle parameters; the objective function includes: a first penalty term negatively correlated with the occlusion area, a reward term positively correlated with the subtitle readability, and a second penalty term negatively correlated with the difference; wherein the occlusion area is determined based on the contour region, the subtitle readability is determined based on the gaze probability distribution and the subtitle priority, and the difference is determined based on the subtitle parameters of the video frame and adjacent video frames preceding the video.

[0238] In some embodiments, the third determining module 4554 is further configured to: determine the target value of each candidate subtitle parameter in the candidate subtitle parameter set under the objective function; and determine the candidate subtitle parameter with the largest target value as the subtitle parameter of the subtitle text in the video frame.

[0239] In some embodiments, the third determining module 4554 is further configured to: determine, based on the candidate subtitle parameters, the candidate subtitle size, candidate transparency, candidate subtitle region, and candidate center point coordinates of the subtitle text in the video frame; determine, based on the gaze probability distribution, the visual focus coordinates of the first object in the video frame; determine, based on the candidate subtitle region, the contour region, and the first penalty item, a first penalty value; determine, based on the candidate subtitle size, the candidate transparency, the subtitle priority, the coordinate distance between the visual focus coordinates and the candidate center point coordinates, and the reward item, a reward value; determine, based on the candidate center point coordinates, the center point coordinates of the subtitle text in adjacent video frames, and the second penalty item, a second penalty value; and determine, based on the first penalty value, the reward value, and the second penalty value, the target value.

[0240] In some embodiments, the third determining module 4554 is further configured to: obtain a third weight value of the first penalty value, a fourth weight value of the reward value, and a fifth weight value of the second penalty value; and use the third weight value, the fourth weight value, and the fifth weight value to perform a weighted summation on the first penalty value, the reward value, and the second penalty value to obtain the target value.

[0241] It should be noted that the description of the apparatus in this application embodiment is similar to the description of the method embodiment described above, and has similar beneficial effects as the method embodiment; therefore, it will not be repeated. For technical details not disclosed in this apparatus embodiment, please refer to the description of the method embodiment of this application for understanding.

[0242] This application provides a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor will execute the video processing method provided in this application.

[0243] This application provides a computer program product including computer-executable instructions stored in a computer-readable storage medium. An electronic device's processor reads the computer-executable instructions from the computer-readable storage medium and executes the instructions, causing the electronic device to perform the video processing method described in this application.

[0244] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0245] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0246] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0247] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0248] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A method of video processing, the method comprising: The method comprises: determining a gaze probability distribution of a first object on each region of a second video frame sequence based on eye movement data of the first object for a first video frame sequence in a target video and the second video frame sequence in the target video; the second video frame sequence is an adjacent video frame sequence located after the first video frame sequence; determining a caption priority of caption text corresponding to the second video frame sequence; the caption priority is used to represent the importance of the caption text in the target video; sequentially performing video frame recognition on each video frame in the second video frame sequence to obtain a contour region of each second object in the video frame; determining a caption parameter of the caption text in the second video frame sequence based on the gaze probability distribution, the caption priority and the contour region; performing caption rendering on the second video frame sequence based on the caption parameter.

2. The method of claim 1, wherein, The eye movement data comprises a gaze point sequence and a pupil diameter sequence; The method comprises: encoding the gaze point sequence to obtain a gaze point feature of the first object for the first video frame sequence; determining a pupil diameter change rate of the first object for the first video frame sequence based on the pupil diameter sequence; extracting a spatio-temporal feature of the second video frame sequence; determining the gaze probability distribution based on the gaze point feature, the pupil diameter change rate and the spatio-temporal feature.

3. The method of claim 2, wherein, The method comprises: performing spatio-temporal block processing on the second video frame sequence to obtain a spatio-temporal block corresponding to the second video frame sequence; extracting a time feature of the second video frame sequence by performing time feature extraction on the spatio-temporal block; the time feature is used to represent dynamic change information between consecutive video frames in the second video frame sequence; extracting a space feature of the second video frame sequence by performing space feature extraction on the spatio-temporal block; the space feature is used to represent static visual information in each video frame in the second video frame sequence; performing first weighted fusion on the time feature and the space feature to obtain the spatio-temporal feature.

4. The method of claim 2, wherein, The method comprises: performing feature mapping on the pupil diameter change rate to obtain an attention weight; performing weighted processing on the gaze point feature based on the attention weight to obtain a weighted feature; performing outer product operation on the spatio-temporal feature and the weighted feature to obtain a space attention matrix; determining the gaze probability distribution based on the space attention matrix and the spatio-temporal feature.

5. The method of claim 4, wherein, The method comprises: Secondly, the spatial attention matrix and the spatio-temporal features are fused by second weighting to obtain a fusion feature; Thirdly, the fusion feature is up-sampled to obtain an up-sampled feature; Fourthly, the up-sampled feature is mapped to obtain the gaze probability distribution.

6. The method of claim 1, wherein, Before determining the eye movement data of the first object for the first sequence of video frames in the target video and a second sequence of video frames in the target video based on the first object, the method further comprises: determining a pupil diameter change rate of the first object for the first sequence of video frames; determining a sliding window length based on the pupil diameter change rate; based on the sliding window length, video frames are cut from the target video starting from a terminal video frame of the first sequence of video frames to obtain the second sequence of video frames.

7. The method of claim 1, wherein, The determination of the caption priority of the caption text corresponding to the second sequence of video frames comprises: encoding the caption text corresponding to the second sequence of video frames to obtain a text encoding vector of the caption text; performing semantic analysis on the text encoding vector to obtain a part-of-speech probability and a sentiment intensity value of each word in the caption text; based on the part-of-speech probability and the sentiment intensity value, determining the caption priority of the caption text.

8. The method of claim 7, wherein, The determination of the caption priority of the caption text based on the part-of-speech probability and the sentiment intensity value comprises: determining the part-of-speech of each word in the caption text based on the part-of-speech probability, and determining the sentiment attribute of each word in the caption text based on the sentiment intensity value; obtaining a first weight value of the part-of-speech and a second weight value of the sentiment attribute; performing weighted summation on the part-of-speech probability and the sentiment intensity using the first weight value and the second weight value to obtain the caption priority.

9. The method of claim 1, wherein, The video frame recognition for each video frame in the second sequence of video frames comprises: target detection is performed for each video frame in the second sequence of video frames to obtain a bounding box of each second object in the video frame and a semantic label of the bounding box; based on the semantic label, a segmentation threshold corresponding to the bounding box is determined; based on the segmentation threshold, image segmentation is performed on an image region corresponding to the bounding box to obtain a contour region of each second object in the video frame.

10. The method of claim 1, wherein, The determination of the caption parameter of the caption text in the second sequence of video frames based on the gaze probability distribution, the caption priority and the contour region comprises: for each video frame in the second sequence of video frames, a candidate caption parameter set of the caption text in the video frame is determined; based on a preset objective function, a caption parameter of the caption text in the video frame is determined from the candidate caption parameter set. The target function comprises a first penalty term negatively related to an occlusion area, a reward term positively related to subtitle readability, and a second penalty term negatively related to a difference degree; wherein the occlusion area is determined based on the contour region, the subtitle readability is determined based on the gaze probability distribution and the subtitle priority, and the difference degree is determined based on the video frame and the subtitle parameters of a neighboring video frame located before the video.

11. The method of claim 10, wherein, The target function is used to determine the subtitle parameters of the subtitle text in the video frame from the candidate subtitle parameter set, comprising: For each candidate subtitle parameter in the candidate subtitle parameter set, determining a target value of the candidate subtitle parameter under the target function; The candidate subtitle parameter with the maximum target value is determined as the subtitle parameter of the subtitle text in the video frame.

12. The method of claim 11, wherein, The determination of the target value of the candidate subtitle parameter under the target function comprises: Based on the candidate subtitle parameter, determining a candidate subtitle size, a candidate transparency, a candidate subtitle region and a candidate center point coordinate of the subtitle text in the video frame; Based on the gaze probability distribution, determining the visual focus coordinates of the first object in the video frame; Based on the candidate subtitle region, the contour region and the first penalty term, determining a first penalty value; Based on the candidate subtitle size, the candidate transparency, the coordinate distance between the visual focus coordinates and the candidate center point coordinates, and the reward term, determining a reward value; Based on the candidate center point coordinates, the center point coordinates of the subtitle text in the neighboring video frame and the second penalty term, determining a second penalty value; Based on the first penalty value, the reward value and the second penalty value, determining the target value.

13. The method of claim 12, wherein, The determination of the target value based on the first penalty value, the reward value and the second penalty value comprises: Obtaining a third weight value of the first penalty value, a fourth weight value of the reward value and a fifth weight value of the second penalty value; Using the third weight value, the fourth weight value and the fifth weight value to weight and sum the first penalty value, the reward value and the second penalty value to obtain the target value.

14. A video processing device, comprising: Comprise: The first determination module is used for determining the gaze probability distribution of the first object on each region of the second video frame sequence based on the eye movement data of the first object for the first video frame sequence in the target video and the second video frame sequence in the target video; the second video frame sequence is a neighboring video frame sequence located after the first video frame sequence; The second determination module is used for determining the subtitle priority of the subtitle text corresponding to the second video frame sequence; the subtitle priority is used to represent the importance of the subtitle text in the target video; The video frame identification module is used for sequentially performing video frame identification on each video frame in the second video frame sequence to obtain the contour region of each second object in the video frame; The third determining module is configured to determine a caption parameter of the caption text in the second video frame sequence based on the gaze probability distribution, the caption priority, and the contour region. The caption rendering module is configured to perform caption rendering on the second video frame sequence based on the caption parameter.

15. An electronic device, comprising: The video processing method comprises the following steps: A memory is configured to store computer executable instructions; A processor is configured to execute the computer executable instructions stored in the memory to implement the video processing method in any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that, A computer readable storage medium stores computer executable instructions, and the computer executable instructions are executed by a processor to implement the video processing method in any one of claims 1 to 13.

17. A computer program product, characterised in that, The computer program product comprises computer executable instructions stored in a computer readable storage medium; The processor of the electronic device reads the computer executable instructions from the computer readable storage medium and executes the computer executable instructions to implement the video processing method in any one of claims 1 to 13.

Images