Subtitle display method, device, electronic device and readable storage medium

By using the subtitle display method based on the audio recognition model during video playback, the problem of difficulty in correspondence between subtitles and video characters is solved, the accurate correspondence between subtitles and video characters is achieved, and the user experience is improved.

CN113992972BActive Publication Date: 2024-11-08VIVO MOBILE COMM CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111267156.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-10-28
Publication Date
2024-11-08
Estimated Expiration
2041-10-28

AI Technical Summary

Technical Problem

Subtitles are difficult to correspond to the characters in the video during video playback, affecting the user's viewing experience.

Method used

By using the trained audio recognition model, during the target video playback, multiple objects are detected and the target object corresponding to the audio data is determined, and the subtitle information is displayed according to the display position of the object. The model is trained based on the second image frame and the corresponding second audio data to ensure accurate identification of the target object in the case of multiple objects.

Benefits of technology

Improves the user's experience of watching videos, ensures that the subtitles correspond accurately to the characters in the video, and avoids missing important content in the picture.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113992972B_ABST
    Figure CN113992972B_ABST
Patent Text Reader

Abstract

The present application discloses a subtitle display method, device, electronic device and readable storage medium, which belongs to the field of terminal technology. The subtitle display method includes: during the playback of a target video, when it is detected that a first image frame of the target video includes multiple first objects, according to an audio recognition model, a target object corresponding to first audio data is determined from multiple first objects, wherein the first audio data corresponds to the first image frame; according to the display position of the target object, subtitle information corresponding to the first audio data is displayed; wherein the audio recognition model is trained based on a second image frame and second audio data corresponding to the second image frame, and the second image frame includes a single first object.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application belongs to the field of terminal technology, and specifically relates to a subtitle display method, device, electronic device and readable storage medium. Background Art

[0002] At present, subtitles have been widely used in various types of multimedia data such as movies, TV series, games, etc. By displaying subtitles, on the one hand, it is convenient for users to appreciate the sound art of the original video, and on the other hand, it is also convenient for users with hearing impairments to watch videos and understand the content of the videos.

[0003] With the development of technology, it is now possible to convert audio data into subtitles in real time through recognition models during video playback. However, due to the difficulty in obtaining effective training data, the subtitles obtained in real time are usually difficult to correspond to the characters in the video, affecting the user's viewing experience. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a subtitle display method, device, electronic device and readable storage medium, which can solve the problem that subtitles are difficult to correspond to characters included in the video, affecting the user's experience of watching the video.

[0005] In a first aspect, an embodiment of the present application provides a subtitle display method, the method comprising:

[0006] During the playback of the target video, when it is detected that a first image frame of the target video includes a plurality of first objects, determining a target object corresponding to first audio data from the plurality of first objects according to an audio recognition model, wherein the first audio data corresponds to the first image frame;

[0007] Displaying subtitle information corresponding to the first audio data according to the display position of the target object;

[0008] The audio recognition model is trained based on the second image frame and second audio data corresponding to the second image frame, and the second image frame includes a single first object.

[0009] In a second aspect, an embodiment of the present application provides a subtitle display device, the device comprising:

[0010] a processing module, configured to determine, during the playback of a target video, a target object corresponding to first audio data from among the multiple first objects when it is detected that a first image frame of the target video includes multiple first objects according to an audio recognition model, wherein the first audio data corresponds to the first image frame;

[0011] A display module, used for displaying subtitle information corresponding to the first audio data according to a display position of the target object;

[0012] The audio recognition model is trained based on the second image frame and second audio data corresponding to the second image frame, and the second image frame includes a single first object.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in the first aspect.

[0014] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0015] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0016] In an embodiment of the present application, during the playback of a target video, for a first image frame including multiple first objects, by using a trained audio recognition model, a target object corresponding to the first audio data can be determined from multiple first objects. Since the audio recognition model is trained based on the second audio data corresponding to the second image frame in the target video, and since the second image frame only includes a single first object, the trained audio recognition model can accurately identify the target object from multiple first objects. Next, according to the display position of the target object, the subtitle information corresponding to the first audio data is displayed, which can facilitate users to understand the target object corresponding to the subtitle information and improve the user's video viewing experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flowchart of a subtitle display method provided by an embodiment of the present application;

[0018] Figure 2 is a schematic diagram of an image frame corresponding to audio data provided by an embodiment of the present application;

[0019] Figure 3 This is a subtitle display schematic diagram provided by an embodiment of the present application;

[0020] Figure 4 is a structural schematic diagram of a subtitle display device provided in an embodiment of the present application;

[0021] Figure 5 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0022] Figure 6 It is a schematic diagram of the hardware structure of another electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0024] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0025] In response to the problems arising from the background technology, the embodiments of the present application provide a subtitle display method, device, electronic device and readable storage medium. During the playback of a target video, for a first image frame including multiple first objects, a target object corresponding to the first audio data can be determined from multiple first objects by using a trained audio recognition model. Since the audio recognition model is trained based on the second audio data corresponding to the second image frame in the target video, and since the second image frame only includes a single first object, the trained audio recognition model can accurately identify the target object from multiple first objects. Next, the subtitle information corresponding to the first audio data is displayed according to the display position of the target object, which can facilitate users to understand the target object corresponding to the subtitle information and improve the user's video viewing experience.

[0026] The following is a detailed description of the subtitle display method, device, electronic device and readable storage medium provided by the embodiments of the present application through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0027] Figure 1 1 is a flow chart of a subtitle display method provided in an embodiment of the present application. The subtitle display method may include steps 110 to 120.

[0028] Step 110 , during the playback of the target video, when it is detected that the first image frame of the target video includes a plurality of first objects, a target object corresponding to the first audio data is determined from the plurality of first objects according to an audio recognition model.

[0029] The first audio data corresponds to the first image frame, the audio recognition model is trained based on the second image frame and the second audio data corresponding to the second image frame, and the second image frame includes a single first object.

[0030] In the embodiment of the present application, the target video may be a TV series, a movie, a short video, a variety show, a video clip, etc., or a live broadcast, or a live broadcast playback, or other videos, which are not limited here. In order to facilitate users to appreciate the various languages ​​included in the target video and understand the original sound in the target video, during the process of playing the target video, the subtitle information displayed on the playback interface may be obtained in real time during the video playback process, or may be the subtitle information cached in the electronic device after the electronic device plays the target video, which is not specifically limited here.

[0031] In some embodiments, there may be one first object or multiple first objects in the target video. For example, some frames in a film or TV series may include only one character, or some frames may include multiple characters at the same time. For another example, in a live broadcast, some frames may include one anchor, or some frames may include multiple anchors. The examples are not listed here.

[0032] In the embodiment of the present application, the first image frame is an image frame corresponding to when the target video includes two or more first objects, and the second image frame is an image frame corresponding to when the target video includes one first object.

[0033] In the embodiment of the present application, the target object corresponding to the audio data can be identified through the audio recognition model. In order to improve the reliability of the audio recognition model, in some embodiments, the second image frame and the second audio data corresponding to the second image frame can be used as training data. Since the second image frame only includes one first object, it can be considered that the audio data corresponding to the second image frame comes from the first object included in the second image.

[0034] Based on this, the electronic device can obtain multiple second image frames in the target video, and the audio data corresponding to each second image frame. Exemplarily, the electronic device can obtain audio feature data in the audio data corresponding to each second image frame, wherein the audio feature data may include, for example, word usage habits, timbre, pitch, and voiceprint information. The audio feature data of the audio data corresponding to each second image frame can be used as training data to train an audio recognition model.

[0035] In an embodiment of the present application, since the audio recognition model is trained based on the audio data of the target video itself and the first object in the image frame as training data, when the first image frame includes multiple first objects, when parsing the first audio data corresponding to the first image frame, it is possible to accurately determine the target object corresponding to the first audio data from the multiple first objects, so that the acquired subtitle information can accurately correspond to the target object among the multiple first objects.

[0036] As a specific embodiment, in order to improve the reliability of the audio recognition model, before parsing the first audio data corresponding to the first image frame according to the audio recognition model and determining the target object corresponding to the first audio data from multiple first objects, the embodiment of the present application may further include the following steps:

[0037] First, a training data set is obtained, wherein the training data set includes a second image frame and second audio data corresponding to the second image frame; next, a preset audio recognition network is trained according to the training data set to obtain an audio recognition model.

[0038] Specifically, the number of first objects included in each image frame in the target video can be confirmed based on image recognition technology. For example, the image recognition technology can identify whether the image frame includes a face. The image recognition technology can be, for example, a preset CNN algorithm, etc., which is not specifically limited here.

[0039] In some embodiments, since the audio feature data of the audio data corresponding to the second image frame may include, for example, word usage habits, timbre, pitch, and voiceprint information, the first object included in the second image frame may correspond one-to-one to the audio feature data corresponding to the second image frame, as training data for the preset audio recognition network. Since the training data comes from the target video itself, not only is the cost of acquiring the training data low, but the reliability of the training data is also high, thereby enabling effective training of the preset audio recognition network.

[0040] In order to improve the accuracy of distinguishing a target object from multiple first objects, in an embodiment of the present application, obtaining a training data set may specifically include the following steps:

[0041] According to the sound utterance time period of the first object in the target video, the target video is segmented to obtain a plurality of image frames corresponding to the audio data of the first object; from the plurality of image frames, a second image frame including a single first object and second audio data corresponding to the second image frame are obtained; the first object in the second image frame is associated with the second audio data to obtain a training data set.

[0042] Specifically, taking the target video as a movie as an example, when the audio data of the movie includes multiple lines of the first object, the sound of the first object in the target video may be, for example, a conversation between different first objects, or a monologue of the first object, etc., which is not specifically limited here.

[0043] When acquiring multiple image frames of the target video, the audio data of the entire movie can be divided according to the lines of each first object, in units of sentences, according to the start time point and the end time point of each line. Accordingly, the target video is segmented according to the start time point and the end time point of each line to obtain image frames corresponding to each line, and then the audio data of each line can be matched one by one with the image frame.

[0044] In some embodiments, according to a preset image recognition algorithm, the number of first objects in each image frame can be determined, and the image frame corresponding to two or more first objects is used as the first image frame, and the image frame corresponding to one first object is used as the second image frame.

[0045] Figure 2 is a schematic diagram of an image frame corresponding to audio data provided by an embodiment of the present application, combined with Figure 2 As shown, according to the sounding time period of the first object, the audio data of the target video can be divided into audio 1 to audio 6 to obtain six sub-audio data. Correspondingly, based on the sounding time period of each first object, the target video can be divided into image frames 1 to image frames 6 to obtain six image frames. Among them, image frame 1, image frame 2, and image frame 3 include the first object A, the first object B, and the first object C respectively, and image frame 1, image frame 2, and image frame 3 can be used as the second image frame. Based on the fact that image frame 1, image frame 2, and image frame 3 correspond to audio 1, audio 2, and audio 3 respectively, therefore, audio 1 can be marked as the sound data of the first object A, audio 2 can be marked as the sound data of the first object B, and audio 3 can be marked as the sound data of the first object C.

[0046] Next, the first object A, the first object B, and the first object C can be associated with the audio 1, the audio 2, and the audio 3, respectively, to obtain a training data set for training a preset audio recognition network. In this way, not only can effective training data be obtained, but also because a large amount of labeled audio data does not need to be prepared in advance, the cost of training data is effectively reduced.

[0047] As a specific example, taking voiceprint as audio feature data, the preset audio recognition network can be, for example, a voiceprint recognition network based on a DNN algorithm, which is not specifically limited here.

[0048] According to the audio recognition model provided in the embodiment of the present application, it is possible to accurately distinguish that the current audio data belongs to a specific target object in the target video, and effective discrimination can be achieved even when multiple first objects speak at the same time.

[0049] As a specific embodiment, continue to combine Figure 2 As shown, image frame 4, image frame 5, and image frame 6 all include two or more first objects, that is, image frame 4, image frame 5, and image frame 6 are all first image frames. Taking image frame 4 as an example, image frame 4 includes first object A, first object B, and first object C. According to the audio recognition model provided in the embodiment of the present application, the audio 4 corresponding to image frame 4 can be analyzed, so as to determine that the audio 4 comes from the specific target object in image frame 4.

[0050] According to the embodiment of the present application, after determining the target object corresponding to the first audio data from the multiple first objects, step 120 may be performed next.

[0051] Step 120: display subtitle information corresponding to the first audio data according to the display position of the target object.

[0052] In some embodiments, subtitle information corresponding to the first audio data may be displayed next to the target object, so that the user can easily understand the specific target object speaking in the first image frame and the specific content said by the target object.

[0053] Exemplarily, the subtitle information corresponding to the first audio can be obtained through a preset audio recognition network. In order to facilitate the user to understand the target object corresponding to the audio data, the subtitles can be displayed near the target object, for example, in the form of bubbles next to the target object. Figure 3 As shown, taking the image frame 4 as an example, by parsing the audio 4, it is recognized that the audio 4 comes from the first object B in the image frame 4, combined with Figure 3 As shown in the subtitle display schematic diagram, the subtitle information corresponding to the audio 4 can be displayed next to the first object B in the form of bubbles.

[0054] As a specific example, when the first audio data includes multiple first objects making sounds at the same time, a speech separation algorithm can be used to separate the first audio data into multiple sub-audio data, and then the sub-audio data can be analyzed by the audio recognition model to determine the target object corresponding to each sub-audio data. If it is identified that multiple target objects have corresponding subtitle information, the subtitle information can be displayed next to each target object, which is not specifically limited here.

[0055] In some embodiments, the display mode of subtitles can be set according to the actual needs of the user, for example, the display color, font, size of subtitles, etc. of the subtitles, which are not specifically limited here.

[0056] In an embodiment of the present application, for a target video being played, when it is detected that a single first object is included in the second image frame of the target video, subtitle information corresponding to the second audio data is obtained; next, the subtitle information is displayed according to the display position of the single first object.

[0057] According to the embodiment of the present application, by displaying corresponding subtitle information according to the location of the first object, the user does not need to focus on the bottom of the screen, avoids missing the wonderful pictures or plots in the picture frame, and improves the user's experience of watching the video.

[0058] In some embodiments, by adding subtitles to each first object in the target video when playing the target video, the first object corresponding to each audio data can be accurately obtained. Next, the first object corresponding to the audio data and the subtitle information of the audio data can be cached in the electronic device, so that when the target video is played again, the subtitle information corresponding to each first object in the target video when it makes a sound can be quickly displayed, thereby improving the user experience.

[0059] It should be noted that the subtitle display method provided in the embodiment of the present application can be executed by a subtitle display device or a control module in the subtitle display device for executing the subtitle display method. In the embodiment of the present application, the subtitle display device executing the subtitle display method is taken as an example to illustrate the subtitle display device provided in the embodiment of the present application.

[0060] Figure 4 is a structural diagram of a subtitle display device provided in an embodiment of the present application, such as Figure 4 As shown, the subtitle display device 400 is applied to a barrage generating device, and the subtitle display device 400 may include: a processing module 410 and a display module 420.

[0061] The processing module 410 is used to determine, during the playback of the target video, a target object corresponding to the first audio data from the multiple first objects according to the audio recognition model when it is detected that the first image frame of the target video includes multiple first objects, wherein the first audio data corresponds to the first image frame;

[0062] The display module 420 is used to display subtitle information corresponding to the first audio data according to the display position of the target object.

[0063] The audio recognition model is trained based on the second image frame and second audio data corresponding to the second image frame, and the second image frame includes a single first object.

[0064] According to the embodiment of the present application, since the audio recognition model is trained based on the second audio data corresponding to the second image frame in the target video, and since the second image frame only includes a single first object, the trained audio recognition model can accurately identify the target object from multiple first objects. Next, according to the display position of the target object, the subtitle information corresponding to the first audio data is displayed, which can facilitate the user to understand the target object corresponding to the subtitle information and improve the user's video viewing experience.

[0065] In some embodiments, the apparatus further comprises:

[0066] The acquisition module is further used to acquire subtitle information corresponding to the second audio data when it is detected that the second image frame of the target video includes a single first object;

[0067] The display module 420 is further configured to display subtitle information according to the display position of the single first object.

[0068] According to the embodiment of the present application, by displaying corresponding subtitle information according to the location of the first object, the user does not need to focus on the bottom of the screen, avoids missing the wonderful pictures or plots in the picture frame, and improves the user's experience of watching the video.

[0069] In some embodiments, the acquisition module is used to acquire a training data set, the training data set including the second image frame and the second audio data corresponding to the second image frame;

[0070] The processing module 410 is further used to train a preset audio recognition network according to the training data set to obtain an audio recognition model.

[0071] In this way, since the training data comes from the target video itself, not only is the cost of obtaining the training data low, but the reliability of the training data is also high, thereby enabling effective training of the preset audio recognition network.

[0072] In some embodiments, the processing module 410 is further configured to segment the target video according to the sound utterance time period of the first object in the target video to obtain a plurality of image frames corresponding to the audio data of the first object;

[0073] The acquisition module is further used to acquire, from the plurality of image frames, a second image frame including a single first object and second audio data corresponding to the second image frame;

[0074] The processing module 410 is further configured to associate the first object in the second image frame with the second audio data to obtain a training data set.

[0075] In this way, not only can effective training data be obtained, but also the cost of training data is effectively reduced because there is no need to prepare a large amount of labeled audio data in advance.

[0076] The subtitle display device in the embodiment of the present application can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc. The non-mobile electronic device can be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0077] The subtitle display device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0078] The subtitle display device provided in the embodiment of the present application can achieve Figures 1 to 4 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0079] Alternatively, if Figure 5 As shown, the embodiment of the present application also provides an electronic device 500, including a processor 501, a memory 502, and a program or instruction stored in the memory 502 and executable on the processor 501. When the program or instruction is executed by the processor 501, each process of the above-mentioned subtitle display method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0080] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0081] Figure 6 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0082] The electronic device 600 includes but is not limited to: a radio frequency unit 601, a network module 602, an audio output unit 603, an input unit 604, a sensor 605, a display unit 606, a user input unit 607, an interface unit 608, a memory 609, and a processor 610.

[0083] Those skilled in the art will appreciate that the electronic device 600 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 610 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 6 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0084] Processor 610 is configured to determine, during playback of a target video, when it is detected that a first image frame of the target video includes a plurality of first objects, according to an audio recognition model, a target object corresponding to first audio data from the plurality of first objects, wherein the first audio data corresponds to the first image frame;

[0085] A display unit 606, configured to display subtitle information corresponding to the first audio data according to a display position of the target object;

[0086] The audio recognition model is trained based on the second image frame and second audio data corresponding to the second image frame, and the second image frame includes a single first object.

[0087] According to the embodiment of the present application, since the audio recognition model is trained based on the second audio data corresponding to the second image frame in the target video, and since the second image frame only includes a single first object, the trained audio recognition model can accurately identify the target object from multiple first objects. Next, according to the display position of the target object, the subtitle information corresponding to the first audio data is displayed, which can facilitate the user to understand the target object corresponding to the subtitle information and improve the user's video viewing experience.

[0088] In some embodiments, the processor 610 is further configured to obtain subtitle information corresponding to the second audio data when it is detected that the second image frame of the target video includes a single first object;

[0089] The display unit 606 is further configured to display subtitle information according to the display position of the single first object.

[0090] According to the embodiment of the present application, by displaying corresponding subtitle information according to the location of the first object, the user does not need to focus on the bottom of the screen, avoids missing the wonderful pictures or plots in the picture frame, and improves the user's experience of watching the video.

[0091] In some embodiments, the processor 610 is configured to obtain a training data set, where the training data set includes a second image frame and second audio data corresponding to the second image frame;

[0092] The processor 610 is further configured to train a preset audio recognition network according to a training data set to obtain an audio recognition model.

[0093] In this way, since the training data comes from the target video itself, not only is the cost of obtaining the training data low, but the reliability of the training data is also high, thereby enabling effective training of the preset audio recognition network.

[0094] In some embodiments, the processor 610 is further configured to segment the target video according to a time period of a first object making a sound in the target video, to obtain a plurality of image frames corresponding to the audio data of the first object;

[0095] The processor 610 is further configured to obtain, from the plurality of image frames, a second image frame including a single first object and second audio data corresponding to the second image frame;

[0096] The processor 610 is further configured to associate the first object in the second image frame with the second audio data to obtain a training data set.

[0097] In this way, not only can effective training data be obtained, but also the cost of training data is effectively reduced because there is no need to prepare a large amount of labeled audio data in advance.

[0098] It should be understood that in the embodiment of the present application, the input unit 604 may include a graphics processor (GPU) 6041 and a microphone 6042, and the graphics processor 6041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 606 may include a display panel 6061, and the display panel 6061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 607 includes a touch panel 6071 and other input devices 6072. The touch panel 6071 is also called a touch screen. The touch panel 6071 may include two parts: a touch detection device and a touch controller. Other input devices 6072 may include but are not limited to a physical keyboard, a function key (such as a volume control button, a switch button, etc.), a trackball, a mouse, and a joystick, which will not be repeated here. The memory 609 can be used to store software programs and various data, including but not limited to applications and operating systems. The processor 610 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, the user interface, and the application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 610.

[0099] The embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned subtitle display method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0100] The processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0101] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned subtitle display method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0102] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0103] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0104] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0105] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A subtitle display method, characterized in that: The method comprises: During the playback of the target video, when it is detected that a first image frame of the target video includes a plurality of first objects, segmenting the target video according to a time period of a sound emission of the first object in the target video to obtain a plurality of image frames corresponding to the audio data of the first object; From the plurality of the image frames, a second image frame including a single first object and a second audio data corresponding to the second image frame are obtained; Establishing an association relationship between the second audio data and the first object in the second image frame to obtain a training data set, wherein the training data set includes the second image frame and the second audio data corresponding to the second image frame; According to the training data set, the preset audio recognition network is trained to obtain an audio recognition model; According to the audio recognition model, a target object corresponding to the first audio data is determined from the plurality of first objects, wherein the first audio data corresponds to the first image frame; According to the display position where the target object is located, the subtitle information corresponding to the first audio data is displayed; When a single first object is detected to be included in the second image frame of the target video, subtitle information corresponding to the second audio data is obtained; The subtitle information is displayed according to the display position of the single first object.

2. A subtitle display device, characterized in that: The device comprises: a processing module, configured to, during the playback of a target video, segment the target video according to a sounding time period of the first object in the target video to obtain a plurality of image frames corresponding to the audio data of the first object when it is detected that the first image frame of the target video includes a plurality of first objects; The acquisition module is also configured to acquire a second image frame including a single first object and a second audio data corresponding to the second image frame from a plurality of the image frames; The processing module is further configured to associate the first object in the second image frame with the second audio data to obtain a training data set, wherein the training data set includes the second image frame and the second audio data corresponding to the second image frame; and train a preset audio recognition network according to the training data set to obtain an audio recognition model; The processing module is further configured to parse first audio data corresponding to the first image frame according to the audio recognition model, and determine a target object corresponding to the first audio data from the plurality of first objects, wherein the first audio data corresponds to the first image frame; The display module is configured to display subtitle information corresponding to the first audio data according to the display position where the target object is located; The acquisition module is also configured to acquire subtitle information corresponding to the second audio data when it is detected that a single first object is included in the second image frame of the target video; The display module is also configured to display the subtitle information according to the display position of the single first object.

3. An electronic device, characterized in that: The method comprises a processor, a memory and a program or instruction stored in the memory and running on the processor, wherein the program or instruction implements the steps of the subtitle display method as claimed in claim 1 when executed by the processor.

4. A readable storage medium, characterized in that: The readable storage medium stores programs or instructions, and the steps of implementing the subtitle display method as claimed in claim 1 when executed by the processor.