Data processing method and device, electronic equipment and storage medium

Abstract

The invention provides a data processing method. The data processing method comprises the steps of obtaining first data; analyzing the first data to obtain a plurality of video frames; outputting to the first object based on the first data; the display position of the target text in the plurality of video frames has a first feature; the target text is a character representation form of the audio data corresponding to the plurality of video frames, the first feature represents that the display position of the target text is an area corresponding to the meaning expression of the audio data, and the area is consistent with the fixation area of the first object.

Description

Technical Field

[0001] The present application relates to but is not limited to the field of data processing technology, and in particular to a data processing method and device, an electronic device and a storage medium. Background Art

[0002] In the related art, when a video is played, the subtitles corresponding to the video are usually displayed at the bottom of the screen. Therefore, when watching the video, the user needs to first move his eyes to the bottom of the screen to view the subtitles, and then move his eyes to the things he actually needs to watch in the video. The above process has problems such as poor flexibility and low efficiency. Summary of the invention

[0003] In view of this, embodiments of the present application at least provide a data processing method and device, an electronic device, and a storage medium.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The present application provides a data processing method, including:

[0006] obtaining first data;

[0007] Parsing the first data to obtain multiple video frames;

[0008] Outputting to a first object based on the first data;

[0009] During the output process, the display position of the target text in the multiple video frames has a first feature; the target text is a textual expression of the audio data corresponding to the multiple video frames, and the first feature characterizes that the display position of the target text is an area corresponding to the meaning expression of the audio data, which is consistent with the gaze area of ​​the first object.

[0010] The present application provides a data processing device, including:

[0011] An acquisition unit, configured to acquire first data;

[0012] A parsing unit, used for parsing the first data to obtain a plurality of video frames;

[0013] An output unit, configured to output to a first object based on the first data;

[0014] During the output process, the display position of the target text in the multiple video frames has a first feature; the target text is a textual expression of the audio data corresponding to the multiple video frames, and the first feature characterizes that the display position of the target text is an area corresponding to the meaning expression of the audio data, which is consistent with the gaze area of ​​the first object.

[0015] An embodiment of the present application provides an electronic device, including a processing unit, wherein:

[0016] A processing unit is used to obtain first data; parse the first data to obtain multiple video frames; output to a first object based on the first data; during the output process, the display position of the target text in the multiple video frames has a first feature; the target text is a textual expression of the audio data corresponding to the multiple video frames, and the first feature represents that the display position of the target text is an area corresponding to the meaning expression of the audio data, and the area is consistent with the gaze area of ​​the first object.

[0017] An embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in any method in the claims are implemented.

[0018] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0020] Figure 1 A schematic diagram of an implementation process of a data processing method provided in an embodiment of the present application Figure 1 ;

[0021] Figure 2 A schematic diagram of a display position of a target text provided in an embodiment of the present application Figure 1 ;

[0022] Figure 3 A schematic diagram of a display position of a target text provided in an embodiment of the present application Figure 2 ;

[0023] Figure 4 A schematic diagram of a display position of a target text provided in an embodiment of the present application Figure 3 ;

[0024] Figure 5 A schematic diagram of a display position of a target text provided in an embodiment of the present application Figure 4 ;

[0025] Figure 6 A schematic diagram of a display position of a target text provided in an embodiment of the present application Figure 5 ;

[0026] Figure 7 A schematic diagram of an implementation process of a data processing method provided in an embodiment of the present application Figure 2 ;

[0027] Figure 8 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0028] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0029] Fig.10 A hardware entity schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application are further elaborated in detail below in conjunction with the drawings and embodiments. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0031] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0032] It should be pointed out that the terms "first\second\third" involved in the embodiments of the present application are merely to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0033] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as the general understanding of those skilled in the art in the field to which the embodiments of the present application belong. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0034] In the related art, when a video is played, the subtitles corresponding to the video are usually displayed at the bottom of the screen. Therefore, when watching the video, the user needs to first move his eyes to the bottom of the screen to view the subtitles, and then move his eyes to the things he actually needs to watch in the video. The above process has problems such as poor flexibility and low efficiency.

[0035] The embodiment of the present application provides a data processing method, which obtains multiple video frames by parsing the first data, and then outputs to the first object based on the first data, wherein, during the output process, the display position of the target text in the multiple video frames is the area corresponding to the meaning expression of the audio data, which is consistent with the gaze area of ​​the first object, and the target text is the text expression of the audio data corresponding to the multiple video frames. In this way, on the one hand, since the display position of the target text in the multiple video frames is the gaze area of ​​the first object, the flexibility of the data processing method is improved; on the other hand, since the display position of the target text is the gaze area of ​​the first object, the first object can view the object of concern and the target text at the same time without switching the line of sight, which improves the accuracy of the data processing method while also improving the intelligence of the data processing method. At the same time, this method can be applied to different electronic devices, improving the versatility of the data processing method.

[0036] The method provided in the embodiment of the present application can be executed by an electronic device, wherein the electronic device can be a laptop, a tablet computer, a desktop computer, a set-top box, a mobile device (for example, a mobile phone, a portable music player, a personal digital assistant, a dedicated messaging device, a portable gaming device) and other types of terminals, and can also be implemented as a server. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0037] Figure 1 A schematic diagram of an implementation process of a data processing method provided in an embodiment of the present application Figure 1 ,like Figure 1 As shown, the method includes steps S101 to S103, wherein:

[0038] Step S101, obtaining first data.

[0039] Here, the first data is data that needs to be output in the device. The first data may include but is not limited to one or more of the following data: video frames, audio data, target text, etc.

[0040] In some implementations, when the first data includes video frames, the number of the video frames is multiple.

[0041] In some implementations, when the first data only includes a video frame, the video frame has corresponding audio data and a target text, and the target text is a textual representation of the audio data corresponding to the video frame.

[0042] In some implementations, when the first data includes video frames and audio data, the audio data is audio data corresponding to the video frames.

[0043] In some implementations, when the first data includes a video frame, audio data, and a target text, the audio data is audio data corresponding to the video frame, and the target text is a textual representation of the audio data corresponding to the video frame.

[0044] In some embodiments, the implementation scenarios of the present application may include, but are not limited to, live broadcast scenarios, recorded broadcast scenarios, and the like. Correspondingly, in a live broadcast scenario, the first data may be data generated in real time, and in a recorded broadcast scenario, the first data may be data that has been recorded. For example, when acquiring the first data, the first data is being recorded synchronously. At this time, the first data is data generated in real time, and the implementation scenario of the present application is a live broadcast scenario. For another example, when acquiring the first data, the first data may be a movie. At this time, the first data is data that has been recorded, and the implementation scenario of the present application is a recorded broadcast scenario.

[0045] The method for obtaining the first data may include but is not limited to: obtaining the first data by receiving data sent by other devices, obtaining the first data by reading data from a device, etc. For example, a large amount of data including the first data is stored in other devices such as cloud devices or servers. By establishing a communication connection between the other devices and the execution device of the present application, the execution device of the present application can receive the first data sent by the other devices. For another example, the first data can be stored in the memory of the execution device of the present application, and the first data can be obtained by reading the data in the memory.

[0046] Step S102: parse the first data to obtain a plurality of video frames.

[0047] Here, a video frame is a single image picture that is the smallest unit in an image animation. A video frame is a still picture, and continuous frames can form an animation. Since the first data may include a video frame, multiple video frames can be obtained by parsing the first data.

[0048] The method for obtaining multiple video frames may include but is not limited to: obtaining multiple video frames by parsing the first data using a video processing library, obtaining multiple video frames by parsing the first data using a cloud service, etc. For example, the video processing library may include but is not limited to: OpenCV, FFmpeg, etc. OpenCV is an open source computer vision and machine learning software library. It provides the functions of reading, displaying and processing video frames. When using OpenCV, the first data can be parsed to obtain multiple video frames by calling the read() function method in a loop. FFmpeg is an open source multimedia framework that provides the functions of recording, converting and streaming audio and video. When using FFmpeg, the first data can be parsed by a command line tool or by calling a function. For another example, a cloud service provider provides a video processing service, and the cloud service provider's interface can be called to upload the first data, and then the first data can be parsed through the cloud service. Cloud service providers may include but are not limited to: AWS, Azure, etc.

[0049] Step S103, output to the first object based on the first data; during the output process, the display position of the target text in the multiple video frames has a first feature; the target text is a textual expression of the audio data corresponding to the multiple video frames, and the first feature represents that the display position of the target text is an area corresponding to the meaning expression of the audio data, and the area is consistent with the gaze area of ​​the first object.

[0050] Here, the first object refers to an object that interacts with the output first data. The first object can be any suitable object, such as a person, a machine, etc.

[0051] The target text is a text representation of the audio data corresponding to the multiple video frames. In some implementations, the target texts corresponding to different multiple video frames may be the same or different.

[0052] In some implementations, the target text may be displayed in the corresponding video frame in the form of a text box, and correspondingly, the display position of the target text may be the coordinates of the text box. A text box is a computer system tool, which is a movable and resizable text or graphic container.

[0053] The display position of the target text refers to the display position of the target text in the corresponding video frame. The display position of the target text may be the area corresponding to the meaning expression of the audio data. The area corresponding to the meaning expression of the audio data may be any suitable area, for example, around the key object in the video frame, around the speaker in the video frame, at the top position of the video frame, etc. The key object refers to the object described by the audio data corresponding to the video frame. The speaker refers to the object that narrates the audio data corresponding to the video frame.

[0054] It should be clarified that in this embodiment, when the video is output, the gaze area of ​​the first object is by default the area corresponding to the meaning expression of the audio data corresponding to the video frame, such as the position of the speaker in the video frame, the position of the key content displayed in the video frame, etc. Of course, the best case is that the gaze area of ​​the first object is the area corresponding to the meaning expression of the audio data. If not, the application can guide the first object to gaze at the area corresponding to the meaning expression of the audio data, so that the first object browses the area corresponding to the meaning expression of each frame of audio data. This embodiment focuses on describing the implementation scheme corresponding to the former case.

[0055] Figure 2 A schematic diagram of a display position of a target text provided in an embodiment of the present application Figure 1 ,like Figure 2 As shown, in the video frame 20 , there is a key object 21 . At this time, the display position 22 of the target text is the area below the key object 21 in the video frame 20 .

[0056] In some implementations, the display position of the target text may be represented by any suitable form, such as Cartesian coordinates, polar coordinates, pixel coordinates, etc.

[0057] The first feature refers to the characteristic of the display position of the target text. In some embodiments, the first feature characterizes that the display position of the target text is an area corresponding to the meaning expression of the audio data, and the area is consistent with the gaze area of ​​the first object. The gaze area of ​​the first object refers to the visual focus area of ​​the first object when interacting with the output first data. The gaze area of ​​the first object can be the position of the audio description object or the object emitting the audio, for example, the center area of ​​the video frame, the upper left corner area of ​​the video frame, etc.

[0058] In some embodiments, the present application assumes that when the first object views the output first data, the first object's gaze area points to the area corresponding to the meaning expression of the audio data. At the same time, since the display position of the target text is the area corresponding to the meaning expression of the audio data, the target text exists in the first object's gaze area.

[0059] In some implementations, the first subject's gaze area may be determined using eye tracking technology, which is a sensor technology that can detect the presence of a person and track what they are looking at in real time.

[0060] In some embodiments, the first data may be output to the first object through a display screen. The display screen is a display tool that displays input information on a screen. The display screen may be any suitable screen, for example, a liquid crystal display (LCD), an organic light-emitting diode display (OLED), a mini LED display, a micro LED display, etc. The method of transmitting the input information to the display screen may include, but is not limited to, a communication protocol, an interface, etc.

[0061] In some implementations, the first data may further include a preset display position, where the preset display position is used to display text of the audio data corresponding to the video frame of the first data.

[0062] In an embodiment of the present application, multiple video frames are obtained by parsing the first data, and then output is performed to the first object based on the first data, wherein, during the output process, the display position of the target text in the multiple video frames is the area corresponding to the meaning expression of the audio data, which is consistent with the gaze area of ​​the first object, and the target text is the text expression of the audio data corresponding to the multiple video frames. In this way, on the one hand, since the display position of the target text in the multiple video frames is the gaze area of ​​the first object, the flexibility of the data processing method is improved; on the other hand, since the display position of the target text is the gaze area of ​​the first object, the first object can view the object of concern and the target text at the same time without switching the line of sight, which improves the accuracy of the data processing method while also improving the intelligence of the data processing method. At the same time, this method can be applied to different electronic devices, improving the versatility of the data processing method.

[0063] In some implementations, the first data includes multiple video frames and audio data, the target text is embedded in corresponding display positions of the multiple video frames, the first data is recorded data, and step S103 includes step S103a, wherein:

[0064] Step S103a: output a plurality of video frames and corresponding audio data to the first object.

[0065] Here, the first data is the data that has been recorded, and accordingly, the implementation scenario of the present application is a recording and broadcasting scenario.

[0066] In some embodiments, since the first data is data that has been recorded, when generating the first data, the display position of the target text in multiple video frames is first determined, and then the target text is embedded in the corresponding video frame according to the determined display position. Therefore, the corresponding display positions in the multiple video frames are embedded with the target text.

[0067] Exemplarily, the first data obtained may be a movie. The movie is synthesized by a data stream corresponding to audio data and a data stream corresponding to a plurality of video frames. When generating a movie, the display position of the subtitles (corresponding to the aforementioned target text) is first determined. The display position of the subtitles may be the position of the object described by the audio data or the position of the object that emits the audio data. Then, according to the determined display position, the subtitles are embedded in a plurality of video frames to obtain data streams corresponding to the plurality of video frames. Therefore, when outputting the movie to the audience (corresponding to the aforementioned first object), a plurality of video frames and corresponding audio data are output, and the corresponding display positions in the plurality of video frames are embedded with subtitles. When watching a movie, the audience will look at the position of the object described by the audio data or the position of the object that emits the audio data, so the audience can view the things that need to be paid attention to and the subtitles at the same time without switching their sight.

[0068] In some embodiments, the first data includes multiple video frames and audio data, and the target text can be displayed in the display position of the target text in the form of hard subtitles. Hard subtitles are also called embedded subtitles. The target text does not have a separate subtitle file, and the display position of the target text cannot be changed.

[0069] In some implementations, when outputting the plurality of video frames and the corresponding audio data to the first object, the plurality of video frames and the corresponding audio data are data that has been completely recorded.

[0070] In some embodiments, multiple video frames may be output to the first object through a display screen, and audio data corresponding to the multiple video frames may be output to the first object through an audio output component. The audio output component may include, but is not limited to, a speaker, a headphone jack, etc. For example, a speaker may convert an electrical signal into a sound wave and output audio data corresponding to the multiple video frames to the first object. For another example, the audio data corresponding to the multiple video frames may be output to the first object through headphones connected to the headphone jack.

[0071] In some embodiments, multiple video frames and corresponding audio data may be output to the first object via a media interface. The media interface may include, but is not limited to, a High Definition Multimedia Interface (HDMI), a DisplayPort (DP), etc. HDMI is a fully digital video and sound transmission interface that can output multiple video frames and corresponding audio data simultaneously. DP is a standardized digital video interface that can output multiple video frames and corresponding audio data simultaneously.

[0072] In some embodiments, when outputting multiple video frames and audio data to the first object, the multiple video frames and audio data can be aligned by using the timestamps corresponding to the multiple video frames and the timestamps corresponding to the audio data, so that the multiple video frames and the corresponding audio data can be output to the first object, so that the multiple video frames and the audio data are output synchronously. The timestamp refers to the total number of seconds from Greenwich Mean Time (i.e., 00:00:00 on January 1, 1970) to the present.

[0073] In an implementation manner of the present application, when the first data is data that has been recorded, multiple video frames and corresponding audio data are output to the first object, wherein the target text is embedded in the corresponding display positions of the multiple video frames, thereby achieving accurate output of video frames with hard subtitles in the recording and broadcasting scenario, broadening the scenario of outputting the first data, and improving the accuracy of outputting the first data.

[0074] In some implementations, the first data includes a plurality of video frames, audio data, and target text, and the first data is data that has been recorded. The method further includes steps S104 to S105, and step S103 includes step S103b, wherein:

[0075] Step S104, parsing the first data to obtain the target text corresponding to each video frame.

[0076] Here, the first data is the data that has been recorded, and correspondingly, the implementation scenario of this application is a recording and broadcasting scenario. The first data includes multiple video frames, audio data and target text, wherein the audio data is the audio data corresponding to the multiple video frames, and the target text is the textual expression of the audio data corresponding to the multiple video frames.

[0077] In some embodiments, when the first data includes multiple video frames, audio data and target text, the target text can be displayed in the preset area in the form of soft subtitles. Soft subtitles are also called built-in subtitles. The target text has a separate subtitle file, the display position of the target text can be changed, and operations such as extraction, editing and deletion can be performed on the target text.

[0078] The method for obtaining the target text corresponding to each video frame may include but is not limited to: obtaining the target text corresponding to each video frame by parsing the first data using a video processing library, obtaining the target text corresponding to each video frame by using MediaToolkit, etc. For example, the video processing library may include but is not limited to: OpenCV, FFmpeg, etc. Through the video processing library, the first data is parsed using a command line tool or a function call to obtain the target text corresponding to each video frame. For another example, MediaToolkit is a NET library that can be used for video processing and has the functions of cutting, resizing, and converting video and audio formats. Through MediaToolkit, the first data can be cut and sliced, so that the first data can be parsed to obtain the target text corresponding to each video frame.

[0079] Step S105, determining the display position of the target text corresponding to each video frame.

[0080] Here, the display position of the target text can be represented by any suitable form, such as Cartesian coordinates, polar coordinates, pixel coordinates, etc. For example, when using Cartesian coordinates, the display position of the target text is usually represented by two-dimensional coordinates (x, y), wherein x represents the horizontal position of the target text, and y represents the vertical position of the target text. For another example, when using polar coordinates, by using (r, θ) to represent the display position of the target text, r represents the distance from the target text to the origin, and θ represents the angle between the target text and the positive x-axis. For another example, when using pixel coordinates, the display position of the target text is usually represented by two-dimensional coordinates (x, y), wherein x represents the horizontal offset position of the target text relative to the upper left corner of the screen, and y represents the vertical offset position of the target text relative to the upper left corner of the screen.

[0081] The method for determining the display position of the target text may include, but is not limited to: determining the display position of the target text by the display position of the speaker, determining the display position of the target text by the display position of the key object, etc. For example, the display position of the speaker may be any suitable position, such as the lower left corner of the video frame, the upper right corner of the video frame, etc. When there is a speaker in a certain video frame, the display position above the speaker may be used as the display position of the target text. For another example, the display position of the key object may be any suitable position, such as the center position of the video frame, the lower right corner of the video frame, etc. When there is a key object in a certain video frame, the display position below the key object may be used as the display position of the target text.

[0082] Step S103b: output multiple video frames and corresponding audio data to the first object, and while outputting each video frame, overlay and display the target text corresponding to the video frame at the determined display position.

[0083] Here, after the display position of the target text is determined, the target text corresponding to the video frame needs to be superimposed and displayed in the corresponding video frame according to the determined position.

[0084] In some embodiments, the present application assumes that when the first object views the output first data, the first object's gaze area points to the area corresponding to the meaning expression of the audio data. At the same time, since the display position of the target text is the area corresponding to the meaning expression of the audio data, the target text exists in the first object's gaze area.

[0085] The target text superimposition method may include but is not limited to: superimposing and displaying the target text corresponding to the video frame at the display position of the target text by using a video processing library, superimposing and displaying the target text corresponding to the video frame at the display position of the target text by a real-time display mechanism, etc. For example, the video processing library may include but is not limited to: OpenCV, FFmpeg, etc. The video processing library may superimpose and display the target text corresponding to the video frame at the display position of the target text by calling a function or a command line tool. For another example, the real-time display mechanism is a mechanism that displays the processing results or added elements (e.g., text) to the user, and superimposes and displays the target text corresponding to the video frame at the display position of the target text by calling the imshow() function.

[0086] Exemplarily, the first data obtained may be a movie. The movie is a complete file synthesized by a data stream corresponding to audio data, a data stream corresponding to subtitles, and a data stream corresponding to a plurality of video frames. When outputting the movie, the display position of the subtitles is first determined. The display position of the subtitles may be the position of the object described by the audio data or the position of the object that emits the audio data. Then, based on the determined display position, the subtitles are embedded in a plurality of video frames. Therefore, when the movie is output to the audience, the corresponding display positions in the plurality of video frames are embedded with subtitles. When watching the movie, the audience will look at the position of the object described by the audio data or the position of the object that emits the audio data. Therefore, the audience can view the things that need to be paid attention to and the subtitles at the same time without switching their sight.

[0087] The process of outputting a plurality of video frames and corresponding audio data to the first object may refer to the specific implementation of step S103a.

[0088] In the implementation mode of the present application, when the first data is the data that has been recorded, the first data is first parsed to obtain the target text corresponding to each video frame, and then the display position of the target text corresponding to each video frame is determined, so as to output multiple video frames and corresponding audio data to the first object, wherein, while outputting each video frame, the target text corresponding to the video frame is superimposed and displayed at the determined display position. The accurate output of video frames with soft subtitles in the recording and broadcasting scenario is achieved, the scenario of outputting the first data is broadened, and the accuracy of outputting the first data is improved.

[0089] In some implementations, the first data includes multiple video frames and audio data, the first data is data generated in real time, the method further includes steps S106 to S108, and step S103 includes step S103c, wherein:

[0090] Step S106, parsing the first data to obtain audio data corresponding to each video frame.

[0091] Here, the first data is data generated in real time, and correspondingly, the implementation scenario of the present application is a live broadcast scenario. The first data includes multiple video frames and audio data, wherein the audio data is audio data corresponding to the multiple video frames.

[0092] In some implementations, the device that generates the first data and the device that outputs the first data may be the same as or different from each other.

[0093] In some implementations, since the first data is data generated in real time, it is necessary to continuously acquire the first data, and then continuously parse the first data to obtain audio data corresponding to each video frame.

[0094] The method for obtaining the audio data corresponding to each video frame may include but is not limited to: obtaining the audio data corresponding to each video frame by parsing the first data using a video processing library, obtaining the audio data corresponding to each video frame by using a graphical interface software, etc. For example, through the video processing library, the first data is parsed using a command line tool or a function call to obtain the audio data corresponding to each video frame. For another example, the graphical interface software may include but is not limited to: HandBrake, VLC MediaPlayer, etc. The graphical interface software has functions such as parsing data conversion formats. Through the graphical interface software, the first data can be parsed to obtain the audio data corresponding to each video frame.

[0095] Step S107, parsing the audio data corresponding to each video frame to obtain the target text corresponding to each video frame.

[0096] Here, when parsing the audio data corresponding to each video frame, the target text corresponding to each video frame may be the same text or different text.

[0097] In some implementations, in a recorded broadcast scenario, the audio data corresponding to each video frame may be parsed in advance, so that when the first data is obtained, the audio data does not need to be parsed again. In a live broadcast scenario, since the first data is generated in real time, the audio data corresponding to each video frame needs to be obtained first, and then the audio data needs to be parsed to obtain the target text corresponding to each video frame.

[0098] The method for obtaining the target text corresponding to each video frame may include but is not limited to: parsing the audio data corresponding to each video frame through speech recognition technology (Automatic Speech Recognition, ASR) to obtain the target text corresponding to each video frame, parsing the audio data corresponding to each video frame through a preset model to obtain the target text corresponding to each video frame, etc. For example, ASR is a technology that uses machine learning or artificial intelligence technology to convert speech into readable text. Through ASR, the audio data corresponding to each video frame can be converted into a target text, thereby obtaining the target text corresponding to each video frame. For another example, the preset model may include but is not limited to: Transformer model, Long Short-Term Memory (LSTM), etc. The Transformer model is a deep neural network model based on the self-attention mechanism, which can parse the audio data corresponding to each video frame to obtain the target text corresponding to each video frame. LSTM is a time-recurrent neural network that is very suitable for processing and predicting sequence data. It can parse the audio data corresponding to each video frame to obtain the target text corresponding to each video frame.

[0099] Step S108, determining the display position of the target text corresponding to each video frame.

[0100] Here, since the target text is displayed in the form of soft subtitles, the display position of the target text may be adjusted so that the display position of the target text in a plurality of video frames has the first feature.

[0101] Methods for determining the display position of the target text may include, but are not limited to: determining the display position of the target text by the display position of the speaker, determining the display position of the target text by the display position of a key object, and the like.

[0102] Step S103c: output multiple video frames and corresponding audio data to the first object, and while outputting each video frame, overlay and display the target text corresponding to the video frame at the determined display position.

[0103] Here, the process of superimposing and displaying the target text corresponding to the video frame at the determined display position may refer to the specific implementation of step S103b.

[0104] The process of outputting a plurality of video frames and corresponding audio data to the first object may refer to the specific implementation of step S103a.

[0105] In the implementation mode of the present application, when the first data is data generated in real time, the first data is first parsed to obtain audio data corresponding to each video frame, and then the audio data corresponding to each video frame is parsed to obtain the target text corresponding to each video frame, and finally the display position of the target text corresponding to each video frame is determined to output multiple video frames and corresponding audio data to the first object, wherein, while outputting each video frame, the target text corresponding to the video frame is superimposed and displayed at the determined display position. The accurate output of video frames with soft subtitles in the live broadcast scene is achieved, the scene of outputting the first data is broadened, and the accuracy of outputting the first data is improved.

[0106] In some implementations, step S108 includes steps S108a to S108b, wherein:

[0107] Step S108a, if there are a second object and at least one third object in the video frame, identify the behavioral characteristics of the second object, if the behavioral characteristics represent that the expression intention of the second object is directed to at least one third object, determine the display position of the target text corresponding to the video frame based on the display position of the at least one third object in the video frame.

[0108] Here, the second object is an object for generating audio data corresponding to a plurality of video frames. The second object may be any suitable object, such as a person, a machine, etc. The third object is an object displayed in the video frame. The third object may be any suitable object, such as a plant, a building, etc. The number of the third objects may be at least one. In some embodiments, the third object does not generate audio. In some embodiments, the second object may be a teacher, and the third object may be an object that the teacher is describing.

[0109] Behavioral characteristics are terms that describe and explain the external manifestations, laws or characteristics of human and animal behavior. The behavioral characteristics of the second object may include but are not limited to: the body language of the second object, the actions performed by the second object, etc. Body language is a non-verbal communication method. The body language of the second object may include but are not limited to: facial expressions, gestures, body postures, etc. The actions performed by the second object may include but are not limited to: reaching out, nodding, etc.

[0110] In some embodiments, the body language of the second object can be obtained by posture estimation. In implementation, posture estimation can be achieved through OpenPose. OpenPose is a popular open source library that can be used for real-time multi-person posture estimation from images. OpenPose can detect 18 key points on the human body, including joints such as the head, elbows, wrists, knees and ankles. When performing head posture estimation, the posture of the head can be calculated by the coordinates of the facial key points. The coordinates of the facial key points may include but are not limited to: the pitch angle of the head relative to the horizontal plane (also known as pitch), the roll angle of the head relative to the vertical plane (also known as roll), and the yaw angle of the head relative to the vertical plane (also known as yaw).

[0111] The expression intention of the second object refers to the purpose or idea conveyed by the behavioral characteristics of the second object. In some embodiments, the expression intention of the second object can be determined by the body language of the second object or the execution action of the second object. For example, when the second object continuously points a finger at a third object within a certain period of time, it can be determined that the behavioral characteristics of the second object represent that the expression intention of the second object is directed to at least one third object.

[0112] In some embodiments, the behavior characteristics of the second object can be continuously determined by direction detection to better analyze the expression intention of the second object. Direction detection can track the movement of an object or a person and determine its direction based on the change of its position over time. In implementation, the key points in each video frame can be detected first, and then the key points can be tracked in consecutive video frames. Direction detection can be achieved by calculating the position change, direction vector, etc. of the key points in the consecutive video frames.

[0113] In some embodiments, when there is at least one object in the video frame, a second object can be determined from the at least one object by means of face detection. The implementation method of face detection may include but is not limited to: implementing face detection by using OpenCV, implementing face detection by using MediaPipe, etc. For example, OpenCV comes with some pre-trained Haar cascade models that can be used to implement face detection. For another example, MediaPipe provides a pre-built graph to implement face detection, which is called FaceDetection. FaceDetection is based on the BlazeFace model, which is a lightweight and efficient face detector.

[0114] In some embodiments, whether the second object and at least one third object exist in the video frame can be detected by object detection. Object detection methods may include but are not limited to: implementing object detection through a single-stage multi-layer target detection model (Single Shot MultiBox Detector, SSD), implementing object detection through YOLO, etc. SDD uses VGG16 as the basic model, and adds a convolutional layer to obtain more feature maps for object detection, with faster speed and accuracy. YOLO is a deep learning framework for object detection. It can detect objects and their locations in an image by viewing the image only once, and can be used to implement object detection.

[0115] The display position of the third object in the video frame refers to the display position of the third object in the corresponding video frame. The display position of the third object in the video frame can be any suitable position, for example, the center position of the video frame, the top position of the video frame, etc.

[0116] In some embodiments, the display position of the target text corresponding to the video frame is determined based on the display position of at least one third object in the video frame. The display position of the target text corresponding to the video frame may be determined in the surrounding area of ​​the display position of the third object in the video frame. The surrounding area of ​​the display position of the third object in the video frame may include, but is not limited to: below the display position of the third object in the video frame, the upper right corner of the display position of the third object in the video frame, etc.

[0117] Exemplarily, in the scenario of online education live broadcast, the first data obtained may include multiple video frames and audio data. If there are a teacher (corresponding to the aforementioned second object) and a dog image (corresponding to the aforementioned third object) in the video frame, it is recognized that the teacher's expression intention is directed to the dog image, and the display position of the subtitles corresponding to the video frame can be determined according to the display position of the dog image in the video frame. The teacher's expression intention is directed to the dog image, which may include but is not limited to at least one of the following: an image of the teacher explaining the dog, an image of the teacher pointing his finger at the dog, an image of the teacher turning his head to look at the dog, etc.

[0118] Figure 3 A schematic diagram of a display position of a target text provided in an embodiment of the present application Figure 2 ,like Figure 3 As shown, in the video frame 30, there are a teacher 31 and a dog image 32. At this time, the behavioral characteristics of the teacher 31 are identified and the behavioral characteristics of the teacher 31 represent that the teacher 31 is explaining the dog image 32. The display position 33 of the subtitle is the upper right corner of the display position of the dog image 32 in the video frame 30.

[0119] Step S108b, if the second object exists in the video frame but the third object does not exist, determine the display position of the target text corresponding to the video frame based on the display position of the second object in the video frame; the second object generates audio, and the third object does not generate audio.

[0120] Here, the display position of the second object in the video frame refers to the display position of the second object in the corresponding video frame. The display position of the second object in the video frame can be any suitable position, for example, the lower left corner position of the video frame, the bottom position of the video frame, etc.

[0121] In some embodiments, the display position of the target text corresponding to the video frame is determined based on the display position of the second object in the video frame. The display position of the target text corresponding to the video frame may be determined around the display position of the second object in the video frame. The area around the display position of the second object in the video frame may include, but is not limited to: above the display position of the second object in the video frame, the upper left corner of the display position of the second object in the video frame, etc.

[0122] Exemplarily, in the scenario of online education live broadcast, the first data obtained may include multiple video frames and audio data. If there is a teacher in the video frame and no third object, the display position of the subtitle corresponding to the video frame can be determined according to the display position of the teacher in the video frame.

[0123] Figure 4 A schematic diagram of a display position of a target text provided in an embodiment of the present application Figure 3 ,like Figure 4 As shown, in the video frame 40 , there is a teacher 41 and no third object. At this time, the display position 42 of the subtitle is above the display position of the teacher 41 in the video frame 40 .

[0124] In some embodiments, the display position of the head of the second object in the video frame can be determined based on the display position of the second object in the video frame, and then the display position of the target text corresponding to the video frame can be set to be near the display position of the head of the second object based on the display position of the head of the second object in the video frame.

[0125] In an embodiment of the present application, the display position of the target text is determined based on the display position of the second object in the video frame or the display position of at least one third object in the video frame through the second object and at least one third object in the video frame. While flexibly determining the display position of the target text, the accuracy of the display position of the target text is also improved.

[0126] In some implementations, step S108 includes steps S108c to S108d, wherein:

[0127] Step S108c, if there is no second object in the video frame but there is at least one third object, identify the expression intention of the audio data corresponding to the video frame; if the expression intention points to at least one third object, determine the display position of the target text corresponding to the video frame based on the display position of the at least one third object in the video frame.

[0128] Here, the expression intention of the audio data refers to the purpose or idea conveyed by the audio data. The expression intention of the audio data may include but is not limited to: describing an object, issuing instructions, etc.

[0129] In some embodiments, the target text corresponding to the audio data may be determined first, and then the expression intent of the audio data may be determined through Natural Language Understanding (NLU). NLU refers to the ability of a computer program to understand, interpret, and generate human language. It involves understanding the structure, meaning, context, and purpose of language, which enables computers to process natural language like humans. When performing NLU, the expression intent of the audio data may be determined through semantic analysis. Semantic analysis is a subfield of NLU that can understand the meaning of words, phrases, and sentences in a text.

[0130] In some embodiments, if the expression intention of the audio data corresponding to the video frame includes describing a third object, it can be determined that the expression intention points to at least one third object, and then the display position of the target text corresponding to the video frame is determined based on the display position of the at least one third object in the video frame.

[0131] In some embodiments, the display position of the target text corresponding to the video frame is determined based on the display position of at least one third object in the video frame, and the display position of the target text corresponding to the video frame may be determined in the surrounding area of ​​the display position of the third object in the video frame. The surrounding area of ​​the display position of the third object in the video frame may include, but is not limited to: below the display position of the third object in the video frame, to the right of the display position of the third object in the video frame, etc.

[0132] Exemplarily, in the scenario of online education live broadcast, the first data obtained may include multiple video frames and audio data, wherein the audio data is the sound of a teacher speaking. If there is no teacher in the video frame but there is an image of a dog, it is recognized that the expression intention of the audio data points to the image of the dog, and the display position of the subtitles corresponding to the video frame can be determined according to the display position of the image of the dog in the video frame. The expression intention of the audio data points to the image of the dog, which may be that the audio data describes the image of the dog.

[0133] Figure 5 A schematic diagram of a display position of a target text provided in an embodiment of the present application Figure 4 ,like Figure 5 As shown, in the video frame 50 , there is no teacher but there is a dog image 51 . At this time, it is recognized that the teacher is describing the dog image 51 , and the display position 52 of the subtitle is above the display position of the dog image 51 in the video frame 50 .

[0134] Step S108d, if there is no second object in the video frame but there is a third object, determine the display position of the target text corresponding to the video frame based on the gaze area or preset area of ​​the first object; the second object generates audio, and the third object does not generate audio.

[0135] Here, the gaze area of ​​the first object may be any suitable area, for example, the center area of ​​the video frame, the upper left corner area of ​​the video frame, etc. In some embodiments, the gaze area of ​​the first object may be determined by eye tracking technology. The preset area may be any suitable area, for example, the bottom area of ​​the video frame, the top area of ​​the video frame, etc.

[0136] In some embodiments, when determining the display position of the target text corresponding to the video frame based on the gaze area or preset area of ​​the first object, at least one coordinate position in the gaze area or preset area of ​​the first object can be obtained, and then any one coordinate is selected from the at least one coordinate position and used as the display position of the target text.

[0137] For example, in the scenario of online education live broadcast, the first data obtained may include multiple video frames and audio data, wherein the audio data is the sound of the teacher speaking. If there is no teacher in the video frame but there is an image of a dog, it is recognized that the teacher is discussing other content with the student (corresponding to the aforementioned first object), and the display position of the subtitle corresponding to the video frame can be determined according to the student's gaze area or preset area.

[0138] Figure 6 A schematic diagram of a display position of a target text provided in an embodiment of the present application Figure 5 ,like Figure 6 As shown, in the video frame 60 , there is no teacher but there is an image 61 of a dog. At this time, the display position 62 of the target text is the top area of ​​the video frame 60 .

[0139] In an embodiment of the present application, the display position of the target text is determined by at least one third object in the video frame based on the display position of the at least one third object in the video frame, the gaze area of ​​the first object or the preset area. While flexibly determining the display position of the target text, the accuracy of the display position of the target text is also improved.

[0140] In some embodiments, the first data includes multiple video frames, audio data, and a preset display position, and the first data is data generated in real time. The method further includes steps S109 to S110, and step S103 includes step S103d, wherein:

[0141] Step S109, parsing the first data to obtain audio data corresponding to each video frame and a preset display position; the preset display position is used to display the text of the audio data corresponding to the video frame.

[0142] Here, the first data is data generated in real time, and correspondingly, the implementation scenario of the present application is a live broadcast scenario. The first data includes multiple video frames, audio data, and a preset display position, wherein the audio data is audio data corresponding to multiple video frames, and the preset display position is used to overlay and display the target text corresponding to the video frame.

[0143] The preset display position may be any suitable position, for example, the center position of the video frame, the upper left corner position of the video frame, etc. In some embodiments, the preset display position may be represented by any suitable form, for example, Cartesian coordinates, polar coordinates, pixel coordinates, etc.

[0144] The method for obtaining the preset display position may include but is not limited to: obtaining the preset display position by parsing the first data through data filtering, obtaining the preset display position through machine learning, etc. For example, the first data may be parsed to obtain the preset display position through regular expressions or other pattern matching techniques in data filtering. For another example, the first data may be parsed to obtain the preset display position through clustering, principal component analysis, and other methods in machine learning.

[0145] Step S110, parsing the audio data corresponding to each video frame to obtain the target text corresponding to each video frame.

[0146] Here, since the first data is data generated in real time, it is necessary to parse the audio data corresponding to each video frame in real time.

[0147] The process of parsing the audio data corresponding to each video frame and obtaining the target text corresponding to each video frame may refer to the specific implementation of step S107.

[0148] Step S103d, outputting a plurality of video frames and corresponding audio data to the first object, and while outputting each video frame, superimposing and displaying a target text corresponding to the video frame at a corresponding preset display position.

[0149] Here, the process of superimposing and displaying the target text corresponding to the video frame at the determined display position may refer to the specific implementation of step S103b.

[0150] The process of outputting a plurality of video frames and corresponding audio data to the first object may refer to the specific implementation of step S103a.

[0151] In some embodiments, the preset display position is related to the resolution of the device. Since there are differences between the resolutions of different devices, when the device generating the first data and the device outputting the first data are different devices, the preset display position needs to be calibrated. The resolution of the device can be any suitable size, for example, 2K, 4K, etc. During implementation, the difference information between the resolution of the device generating the first data and the resolution of the device outputting the first data can be determined, and then the preset display position is calibrated based on the difference information. In some embodiments, the difference information between the resolution of the device generating the first data and the resolution of the device outputting the first data can be represented by a ratio.

[0152] Exemplarily, the resolution of the device generating the first data is 2K, and the resolution of the device outputting the first data is 4K. Correspondingly, the difference information between the resolution of the device generating the first data and the resolution of the device outputting the first data is twice, and accordingly, the coordinate value of the preset display position is doubled to calibrate the preset display position, and then the target text corresponding to the video frame is superimposed and displayed based on the calibrated preset display position.

[0153] In the implementation mode of the present application, when the first data is data generated in real time, the first data is first parsed to obtain the audio data and preset display position corresponding to each video frame, and then the audio data corresponding to each video frame is parsed to obtain the target text corresponding to each video frame, so as to output multiple video frames and corresponding audio data to the first object, wherein, while outputting each video frame, the target text corresponding to the video frame is superimposed and displayed at the corresponding preset display position. The accurate output of video frames with soft subtitles in the live broadcast scene is achieved, the scene of outputting the first data is broadened, and the accuracy of outputting the first data is improved.

[0154] The following describes the application of the data processing method provided in the embodiment of the present application in actual scenarios, taking online education live broadcast as an example.

[0155] In the related art, when a video is played, the subtitles corresponding to the video are usually displayed at the bottom of the screen. Therefore, when watching the video, the user needs to first move his eyes to the bottom of the screen to view the subtitles, and then move his eyes to the things he actually needs to watch in the video. The above process has problems such as poor flexibility and low efficiency.

[0156] The embodiment of the present application provides a data processing method, which obtains multiple video frames by parsing the first data, and then outputs to the first object based on the first data, wherein, in the process of outputting, the display position of the target text in the multiple video frames is the gaze area of ​​the first object, and the target text is the text expression of the audio data corresponding to the multiple video frames. In this way, on the one hand, since the display position of the target text in the multiple video frames is the area corresponding to the meaning expression of the audio data, the area is consistent with the gaze area of ​​the first object, and the display position of the target text can be dynamically adjusted according to the gaze area of ​​the first object, thereby improving the flexibility of the data processing method; on the other hand, since the display position of the target text is the gaze area of ​​the first object, the first object can view the object of interest and the target text at the same time without switching the line of sight, thereby improving the accuracy of the data processing method and the intelligence of the data processing method. At the same time, the method can be applied to different electronic devices, thereby improving the versatility of the data processing method.

[0157] Figure 7 A schematic diagram of an implementation process of a data processing method provided in an embodiment of the present application Figure 2 ,like Figure 7 As shown, the method includes steps S201 to S207, wherein:

[0158] Step S201, obtaining first data;

[0159] Here, the first data includes a plurality of video frames and audio data.

[0160] Step S202, parsing the first data to obtain a plurality of video frames and audio data corresponding to the plurality of video frames;

[0161] Step S203, parsing the audio data corresponding to each video frame to obtain the subtitles corresponding to each video frame (corresponding to the aforementioned target text);

[0162] Step S204, if there is a focused object (corresponding to the aforementioned third object) in the video frame, determine the display position of the subtitles corresponding to the video frame based on the display position of the focused object in the video frame;

[0163] Step S205, if there is a teacher (corresponding to the aforementioned second object) in the video frame and there is no object pointed at by the focus, the display position of the subtitles corresponding to the video frame is determined based on the display position of the teacher in the video frame;

[0164] Step S206, if there is no teacher and no focused object in the video frame, the display position of the subtitles corresponding to the video frame is determined based on the gaze area or the preset area of ​​the student (corresponding to the aforementioned first object);

[0165] Step S207, outputting multiple video frames and audio data corresponding to the multiple video frames to the students, while outputting each video frame, overlaying and displaying the subtitles corresponding to the video frame at the determined display position of the subtitles.

[0166] Based on the above embodiments, the present application also provides a data processing device. Figure 8 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application is shown in FIG. Figure 8 As shown, the data processing device 800 includes an acquisition unit 801, a parsing unit 802 and an output unit 803, wherein:

[0167] An acquisition unit 801 is used to acquire first data;

[0168] A parsing unit 802, configured to parse the first data to obtain a plurality of video frames;

[0169] An output unit 803, configured to output to a first object based on the first data;

[0170] During the output process, the display position of the target text in the multiple video frames has a first feature; the target text is a textual expression of the audio data corresponding to the multiple video frames, and the first feature characterizes that the display position of the target text is an area corresponding to the meaning expression of the audio data, which is consistent with the gaze area of ​​the first object.

[0171] In some embodiments, the first data includes multiple video frames and audio data, the target text is embedded in the corresponding display positions of the multiple video frames, the first data is the data that has been recorded, and the output unit 803 is also used to output multiple video frames and corresponding audio data to the first object.

[0172] In some embodiments, the first data includes multiple video frames, audio data and target text, and the first data is data that has been recorded. The data processing device 800 also includes a processing unit, wherein the processing unit is used to parse the first data to obtain the target text corresponding to each video frame; determine the display position of the target text corresponding to each video frame; the output unit 803 is also used to output multiple video frames and corresponding audio data to the first object, and while outputting each video frame, the target text corresponding to the video frame is superimposed and displayed at the determined display position.

[0173] In some embodiments, the first data includes multiple video frames and audio data, and the first data is data generated in real time. The data processing device 800 also includes a processing unit, wherein the processing unit is used to parse the first data to obtain audio data corresponding to each video frame; parse the audio data corresponding to each video frame to obtain the target text corresponding to each video frame; determine the display position of the target text corresponding to each video frame; the output unit 803 is also used to output multiple video frames and corresponding audio data to the first object, and while outputting each video frame, the target text corresponding to the video frame is superimposed and displayed at the determined display position.

[0174] In some embodiments, the processing unit is further used to, if there are a second object and at least one third object in the video frame, identify behavioral characteristics of the second object, and determine the display position of a target text corresponding to the video frame based on the display position of the at least one third object in the video frame if the behavioral characteristics characterize that the expression intention of the second object is directed to the at least one third object; if there is a second object but no third object in the video frame, determine the display position of the target text corresponding to the video frame based on the display position of the second object in the video frame; the second object generates audio, and the third object does not generate audio.

[0175] In some embodiments, the processing unit is further used to identify the expression intention of the audio data corresponding to the video frame if there is no second object in the video frame but there is at least one third object, and if the expression intention points to at least one third object, determine the display position of the target text corresponding to the video frame based on the display position of the at least one third object in the video frame; if there is no second object in the video frame but there is a third object, determine the display position of the target text corresponding to the video frame based on the gaze area or preset area of ​​the first object; the second object generates audio, and the third object does not generate audio.

[0176] In some embodiments, the first data includes multiple video frames, audio data and a preset display position. The first data is data generated in real time. The data processing device 800 also includes a processing unit, wherein the processing unit is used to parse the first data to obtain audio data corresponding to each video frame and a preset display position; the preset display position is used to display the text of the audio data corresponding to the video frame; the audio data corresponding to each video frame is parsed to obtain the target text corresponding to each video frame; the output unit 803 is also used to output multiple video frames and corresponding audio data to the first object, and while outputting each video frame, the target text corresponding to the video frame is superimposed and displayed at the corresponding preset display position.

[0177] Based on the above embodiments, the present application also provides an electronic device, Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Fig. 9As shown, the electronic device 900 includes a processing unit 901, wherein:

[0178] Processing unit 901 is used to obtain first data; parse the first data to obtain multiple video frames; output to a first object based on the first data; during the output process, the display position of the target text in the multiple video frames has a first feature; the target text is a textual expression of the audio data corresponding to the multiple video frames, and the first feature represents that the display position of the target text is an area corresponding to the meaning expression of the audio data, and the area is consistent with the gaze area of ​​the first object.

[0179] Here, the first data may include but is not limited to one or more of the following data: video frames, audio data, target text, etc.

[0180] In some implementations, when the first data includes video frames and audio data, the audio data is audio data corresponding to the video frames.

[0181] In some implementations, when the first data includes a video frame, audio data, and a target text, the audio data is audio data corresponding to the video frame, and the target text is a textual representation of the audio data corresponding to the video frame.

[0182] The method for obtaining the multiple video frames may include but is not limited to: obtaining the multiple video frames by parsing the first data using a video processing library, obtaining the multiple video frames by parsing the first data using a cloud service, etc. The video processing library may include but is not limited to: OpenCV, FFmpeg, etc. The cloud service provider may include but is not limited to: AWS, Azure, etc.

[0183] The first object may be any suitable object, for example, a person, a machine, etc. The target text is a text representation of the audio data corresponding to the plurality of video frames.

[0184] In some implementations, the target texts corresponding to different video frames may be the same or different.

[0185] The display position of the target text may be any appropriate position, for example, around a key object in a video frame, around a speaker in a video frame, at the top of a video frame, etc.

[0186] The gaze area of ​​the first object may be any appropriate area, for example, the central area of ​​the video frame, the upper left corner area of ​​the video frame, etc.

[0187] In some embodiments, the target text may be displayed in the corresponding video frame in the form of a text box, and correspondingly, the display position of the target text may be the coordinates of the text box. In some embodiments, the target text may be output to the first object based on the first data through a display screen.

[0188] In an embodiment of the present application, the electronic device obtains multiple video frames by parsing the obtained first data, and then outputs to the first object based on the first data, wherein, during the output process, the display position of the target text in the multiple video frames is the gaze area of ​​the first object, and the target text is the textual expression of the audio data corresponding to the multiple video frames. In this way, on the one hand, since the display position of the target text in the multiple video frames is the area corresponding to the meaning expression of the audio data, which is consistent with the gaze area of ​​the first object, the display position of the target text can be dynamically adjusted according to the gaze area of ​​the first object, thereby improving the flexibility of the data processing method; on the other hand, since the display position of the target text is the gaze area of ​​the first object, the first object can view the object of concern and the target text at the same time without switching the line of sight, thereby improving the accuracy of the data processing method and the intelligence of the data processing method.

[0189] The description of the electronic device embodiment above is similar to the description of the method embodiment above, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the electronic device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0190] It should be noted here that the description of the various embodiments above tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. The description of the above device, storage medium, computer program and computer program product embodiments is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the embodiments of the device, storage medium, computer program and computer program product of this application, please refer to the description of the method embodiment of this application for understanding.

[0191] It should be noted that in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium, including a number of instructions to enable an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0192] An embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor implements the above method when executing the computer program.

[0193] The embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and the computer program implements the above method when executed by a processor. The computer-readable storage medium may be transient or non-transient.

[0194] The embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, some or all of the steps in the above method are implemented. The computer program product can be implemented specifically by hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK) and the like.

[0195] It should be noted that Fig.10 A schematic diagram of a hardware entity of an electronic device provided in an embodiment of the present application is shown in the figure. As shown in the figure, the hardware entity of the electronic device 1000 includes: a processor 1001, a communication interface 1002 and a memory 1003, wherein:

[0196] The processor 1001 generally controls the overall operation of the electronic device 1000 .

[0197] The communication interface 1002 enables the electronic device to communicate with other terminals or servers through a network.

[0198] The memory 1003 is configured to store instructions and applications executable by the processor 1001, and can also cache data to be processed or processed by the processor 1001 and each module in the electronic device 1000 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented by flash memory (FLASH) or random access memory (Random Access Memory, RAM). Data transmission can be performed between the processor 1001, the communication interface 1002, and the memory 1003 through the bus 1004.

[0199] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0200] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the serial number of each step / process mentioned above does not mean the order of execution, and the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The serial numbers of the embodiments of the present application mentioned above are for description only and do not represent the advantages and disadvantages of the embodiments.

[0201] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0202] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0203] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0204] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0205] A person of ordinary skill in the art can understand that: all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, read-only memories, magnetic disks or optical disks.

[0206] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0207] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A data processing method, the method comprising: obtaining first data; Parsing the first data to obtain multiple video frames; Outputting to a first object based on the first data; During the output process, the display position of the target text in the plurality of video frames has a first feature; The target text is a textual representation of the audio data corresponding to the multiple video frames, and the first feature characterizes that the display position of the target text is an area corresponding to the meaning expression of the audio data, and the area is consistent with the gaze area of ​​the first object.

2. The method according to claim 1, wherein the first data comprises the plurality of video frames and audio data, the target text is embedded in corresponding display positions in the plurality of video frames, and the first data is data that has been recorded; The outputting to the first object based on the first data includes: The plurality of video frames and corresponding audio data are output to the first object.

3. The method according to claim 1, wherein the first data includes the plurality of video frames, audio data and target text, and the first data is data that has been recorded, and the method further includes: Parsing the first data to obtain target text corresponding to each video frame; Determine the display position of the target text corresponding to each video frame; The outputting to the first object based on the first data includes: The multiple video frames and the corresponding audio data are output to the first object, and while each video frame is output, a target text corresponding to the video frame is superimposed and displayed at the determined display position.

4. The method according to claim 1, wherein the first data comprises the plurality of video frames and audio data, and the first data is data generated in real time, and the method further comprises: Parsing the first data to obtain audio data corresponding to each video frame; Parse the audio data corresponding to each video frame to obtain the target text corresponding to each video frame; Determine the display position of the target text corresponding to each video frame; The outputting to the first object based on the first data includes: The multiple video frames and the corresponding audio data are output to the first object, and while each video frame is output, a target text corresponding to the video frame is superimposed and displayed at the determined display position.

5. The method according to claim 4, wherein determining the display position of the target text corresponding to each video frame comprises: If there is a second object and at least one third object in the video frame, identifying a behavior feature of the second object, and if the behavior feature indicates that the expression intention of the second object is directed to at least one third object, determining a display position of a target text corresponding to the video frame based on a display position of at least one third object in the video frame; If the second object exists in the video frame but the third object does not exist, determining a display position of the target text corresponding to the video frame based on a display position of the second object in the video frame; The second object generates audio, and the third object does not generate audio.

6. The method according to claim 4, wherein determining the display position of the target text corresponding to each video frame comprises: If the second object does not exist in the video frame but at least one third object exists, identifying the expression intention of the audio data corresponding to the video frame, and if the expression intention points to at least one of the third objects, determining the display position of the target text corresponding to the video frame based on the display position of at least one of the third objects in the video frame; If the second object does not exist in the video frame but the third object exists, determining a display position of the target text corresponding to the video frame based on the gaze area or the preset area of ​​the first object; The second object generates audio, and the third object does not generate audio.

7. The method according to claim 1, wherein the first data includes the plurality of video frames, audio data and a preset display position, and the first data is data generated in real time, and the method further comprises: Parsing the first data to obtain audio data corresponding to each video frame and a preset display position; The preset display position is used to display the text of the audio data corresponding to the video frame; Parse the audio data corresponding to each video frame to obtain the target text corresponding to each video frame; The outputting to the first object based on the first data includes: The multiple video frames and the corresponding audio data are output to the first object, and while each video frame is output, a target text corresponding to the video frame is superimposed and displayed at a corresponding preset display position.

8. A data processing device, comprising: An acquisition unit, configured to acquire first data; A parsing unit, used for parsing the first data to obtain a plurality of video frames; an output unit, configured to output to a first object based on the first data; During the output process, the display position of the target text in the plurality of video frames has a first feature; The target text is a textual representation of the audio data corresponding to the multiple video frames, and the first feature characterizes that the display position of the target text is an area corresponding to the meaning expression of the audio data, and the area is consistent with the gaze area of ​​the first object.

9. An electronic device, comprising a processing unit, wherein: The processing unit is used to obtain first data; parse the first data to obtain multiple video frames; output to a first object based on the first data; during the output process, the display position of the target text in the multiple video frames has a first feature; The target text is a textual representation of the audio data corresponding to the multiple video frames, and the first feature characterizes that the display position of the target text is an area corresponding to the meaning expression of the audio data, and the area is consistent with the gaze area of ​​the first object.

10. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

Images