Information display method, device, computer equipment and computer readable storage medium

Abstract

The present application provides an information display method, device, equipment and computer-readable storage medium; the method includes: during video playback, receiving an object recognition instruction for a target object in a video frame image corresponding to a current playback time point; in response to the object recognition instruction, obtaining a video frame image corresponding to at least one target playback time point associated with the current playback time point; from the obtained video frame images and the video frame images corresponding to the current playback time point, screening out video frame images that meet video recognition conditions as candidate video frame images; performing target object recognition on the candidate video frame images to obtain a recognition result, and based on the recognition result, displaying the object information of the target object. Through the present application, the accuracy of object information display can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an information display method, device, computer equipment and computer-readable storage medium. Background Art

[0002] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making. With the continuous development of artificial intelligence technology, artificial intelligence has been increasingly applied to video playback or processing.

[0003] When watching videos, users often need to know detailed information about the people in the video. However, in related technologies, object recognition is usually performed based on the currently presented picture, and then the object information in the currently presented picture is displayed to the user. However, object recognition is only performed based on the currently presented picture, which has the problem of low recognition accuracy. Summary of the invention

[0004] The embodiments of the present application provide an information display method, apparatus, computer device and computer-readable storage medium, which can improve the accuracy of object information display.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application provides an information display method, including:

[0007] During video playback, an object recognition instruction is received for a target object in a video frame image corresponding to a current playback time point;

[0008] In response to the object recognition instruction, acquiring a video frame image corresponding to at least one target playback time point associated with the current playback time point;

[0009] Filtering out video frame images that meet the video recognition condition from the acquired video frame images and the video frame images corresponding to the current playback time point as candidate video frame images;

[0010] The target object is identified on the candidate video frame image to obtain an identification result, and based on the identification result, object information of the target object is displayed.

[0011] The present application provides an information display method, including:

[0012] During video playback, an object recognition instruction sent by a control device is received, wherein the object recognition instruction is used to instruct recognition of a target object in a video frame image corresponding to a current playback time point;

[0013] In response to the object recognition instruction, displaying object information of the target object;

[0014] The object information is obtained by performing target object recognition on video frame images corresponding to the current playback time point and video frame images corresponding to at least one target playback time point associated with the current playback time point, which meet the video recognition conditions.

[0015] The present application provides an information display device, including:

[0016] A first receiving module is used to receive an object recognition instruction for a target object in a video frame image corresponding to a current playback time point during video playback;

[0017] An acquisition module, configured to acquire, in response to the object recognition instruction, a video frame image corresponding to at least one target playback time point associated with the current playback time point;

[0018] A screening module is used to screen out video frame images that meet the video recognition conditions from the acquired video frame images and the video frame images corresponding to the current playback time point as candidate video frame images;

[0019] The first display module is used to perform target object recognition on the candidate video frame image, obtain a recognition result, and display object information of the target object based on the recognition result.

[0020] In the above solution, the first receiving module is further used to receive an object recognition instruction sent by the control device for a target object in a video frame image corresponding to a current playback time point;

[0021] The object recognition instruction is triggered based on an object recognition button on the control device.

[0022] In the above solution, the first receiving module is further used to respond to the object recognition start instruction, and when the video frame image corresponding to the current playback time point contains at least two objects, present a target object selection interface;

[0023] A selection operation triggered by a selection interface based on the target object is received, and the object corresponding to the selection operation is used as the target object.

[0024] In the above solution, when the at least one target playback time point includes a first number of backward playback time points and a second number of forward playback time points, the first receiving module is further used to obtain a target time interval;

[0025] Based on the target time interval, starting from the target playback time point and moving backwards, every target time interval, obtaining a backward playback time point as the target playback time point until a first number of backward playback time points are obtained;

[0026] Based on the target time interval, starting from the target playback time point and moving forward, a forward playback time point is acquired as the target playback time point every target time interval until a second number of forward playback time points are acquired.

[0027] In the above solution, the acquisition module is further used to determine the playback time period including the current playback time point;

[0028] At least one playback time point is selected from the playback time period as a target playback time point.

[0029] In the above solution, the acquisition module is also used to determine the similarity between each acquired video frame image and the video frame image corresponding to the current playback time point;

[0030] Screening out video frame images whose similarity reaches a similarity threshold from the acquired video frame images;

[0031] From the video frame images whose similarity reaches the similarity threshold and the video frame images corresponding to the current playback time point, the video frame images that meet the video recognition condition are screened out as candidate video frame images.

[0032] In the above solution, the screening module is further used to determine the clarity of the target object in each acquired video frame image and the clarity of the target object in the video frame image corresponding to the current playback time point;

[0033] The acquired video frame images and the video frame images corresponding to the current playback time point are sorted in descending order of face clarity to obtain a first video frame image sequence;

[0034] Starting from the first video frame image of the first video frame image sequence, a target number of video frame images are acquired as candidate video frame images.

[0035] In the above scheme, the screening module is also used to determine the face area size of the target object in each acquired video frame image, and the face area size of the video frame image corresponding to the current playback time point;

[0036] The acquired video frame images and the video frame images corresponding to the current playback time point are sorted in descending order of face region size to obtain a second video frame image sequence;

[0037] Starting from the first video frame image of the second video frame image sequence, a target number of video frame images are acquired as candidate video frame images.

[0038] In the above scheme, the screening module is also used to determine the face area size of the target object in each acquired video frame image, and the face area size of the video frame image corresponding to the current playback time point;

[0039] Get the face area size threshold;

[0040] From the acquired video frame images and the video frame images corresponding to the current playback time point, video frame images whose face region sizes reach the face region size threshold are screened out as candidate video frame images.

[0041] In the above scheme, the first display module is further used to perform face recognition on the target image in the candidate video frame image to obtain at least two candidate recognition results corresponding to the target object and corresponding credibility;

[0042] Get a list of objects corresponding to the video;

[0043] Based on the importance of each object in the object list in the video, weighting the credibility corresponding to each candidate recognition result to obtain a weighted credibility corresponding to each candidate recognition result;

[0044] The candidate recognition result with the highest weighted credibility is taken as the recognition result of the target object.

[0045] In the above solution, the first display module is further used to obtain the duration of each object in the object list appearing in the video;

[0046] Determine the weight of each object according to the duration of each object appearing in the video.

[0047] Based on the weight of the object corresponding to each candidate recognition result, the credibility corresponding to the candidate object is weighted.

[0048] In the above scheme, the first display module is further used to perform face recognition on the target image in the candidate video frame image to obtain at least two candidate recognition results corresponding to the target object and corresponding credibility;

[0049] Obtaining an object list corresponding to the video, and matching each candidate recognition result with an object in the object list;

[0050] Filtering out a candidate recognition result having a matching object from at least two candidate recognition results;

[0051] From the screened candidate recognition results, determine the candidate recognition result with the highest credibility as the recognition result of the target object.

[0052] The present application provides an information display device, including:

[0053] A second receiving module is used to receive an object recognition instruction sent by a control device during video playback, wherein the object recognition instruction is used to instruct to recognize a target object in a video frame image corresponding to a current playback time point;

[0054] A second display module, configured to display object information of the target object in response to the object recognition instruction;

[0055] The object information is obtained by performing target object recognition on video frame images corresponding to the current playback time point and video frame images corresponding to at least one target playback time point associated with the current playback time point, which meet the video recognition conditions.

[0056] An embodiment of the present application provides a computer device, including:

[0057] A memory for storing executable instructions;

[0058] The processor is used to implement the information display method provided in the embodiment of the present application when executing the executable instructions stored in the memory.

[0059] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for causing a processor to execute and implement the information display method provided in the embodiment of the present application.

[0060] The embodiments of the present application have the following beneficial effects:

[0061] The present application receives an object recognition instruction for a target object in a video frame image corresponding to a current playback time point during video playback; in response to the object recognition instruction, obtains a video frame image corresponding to at least one target playback time point associated with the current playback time point; selects video frame images that meet video recognition conditions from the obtained video frame images and the video frame images corresponding to the current playback time point as candidate video frame images; recognizes the target object based on the candidate video frame images to obtain a recognition result, and displays object information of the target object based on the recognition result; in this way, since the present application not only obtains the video frame image corresponding to the current playback time point, but also obtains the video frame image corresponding to at least one target playback time point associated with the current playback time point, the selected candidate video frame images are more conducive to object recognition, thereby improving the accuracy of target object recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1A This is an optional architecture diagram of the information display system provided in the embodiment of the present application;

[0063] Figure 1B This is an optional architecture diagram of the information display system provided in the embodiment of the present application;

[0064] Figure 2 It is a flowchart of the information display method provided in the embodiment of the present application;

[0065] Figure 3 is a schematic diagram of a target object selection interface provided in an embodiment of the present application;

[0066] Figure 4 is a schematic diagram of a target object selection interface provided in an embodiment of the present application;

[0067] Figure 5 is a schematic diagram of presenting object information of a target object provided by an embodiment of the present invention;

[0068] Figure 6 It is a flowchart of the information display method provided in the embodiment of the present application;

[0069] Figure 7 It is a flowchart of the information display method provided in the embodiment of the present application;

[0070] Figure 8 It is a flowchart of the information display method provided in the embodiment of the present application;

[0071] Fig. 9 It is a flowchart of the information display method provided in the embodiment of the present application;

[0072] Fig.10is a flowchart of a process for acquiring a candidate video frame image provided by an embodiment of the present application;

[0073] Fig.11 is an example diagram of a face recognition interface provided in an embodiment of the present application;

[0074] Fig.12 is a flowchart of an object recognition process provided by an embodiment of the present application;

[0075] Fig.13 is a schematic diagram of an interface for displaying object information provided in an embodiment of the present application;

[0076] Fig.14 is a structural schematic diagram of an information display device provided in an embodiment of the present application;

[0077] Fig.15 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0078] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0079] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0080] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0081] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0082] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0083] 1) In response, it is used to indicate the conditions or states on which the executed operations depend. When the dependent conditions or states are met, one or more operations executed may be in real time or have a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations executed are executed.

[0084] Based on the above explanations of the nouns and terms involved in the embodiments of the present application, the information display system provided by the embodiments of the present application is described below. Figure 1A , Figure 1A This is an optional architecture diagram of the information display system provided in the embodiment of the present application. To support an exemplary application, the terminal (exemplarily showing terminal 400-1 and terminal 400-2) is connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two. In actual implementation, the terminal is provided with a client, such as a video client, a browser client, a news client, an education client, etc., for playing videos.

[0085] The terminal (such as 400-1) is used to receive an object recognition instruction for a target object in a video frame image corresponding to a current playback time point during video playback; in response to the object recognition instruction, obtain a video frame image corresponding to at least one target playback time point associated with the current playback time point; select a video frame image that meets a video recognition condition from the obtained video frame images and the video frame images corresponding to the current playback time point as a candidate video frame image; and send an object recognition request carrying the candidate video frame image to the server 200;

[0086] The object recognition instruction here may be triggered based on an object recognition function item presented by the terminal, or may be triggered based on an object recognition button on the terminal. The triggering method of the object recognition instruction is not limited here;

[0087] The server 200 is used to identify the target object on the candidate video frame image, obtain the identification result of the target object, obtain the object information of the target object based on the identification result, and send the object information of the target object to the terminal;

[0088] The terminal is used to display the object information of the target object.

[0089] In some embodiments, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDN, Content Delivery Network), and big data and artificial intelligence platforms. The terminal may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a TV terminal, a car terminal, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0090] See also Figure 1B , Figure 1B It is an optional architectural diagram of the information display system provided in an embodiment of the present application. To support an exemplary application, a control device (such as control device 500-1 or control device 500-2) establishes a communication connection with a terminal 600 to achieve data transmission. For example, when the terminal is a TV terminal, a communication connection with the TV terminal is established through a radio frequency module included in the TV terminal to achieve data transmission through wireless communication.

[0091] In practical applications, the terminal may be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, TV terminal (including Internet TV all-in-one, set-top box + TV, etc.), vehicle-mounted terminal, etc. The control device includes but is not limited to a keyboard, remote controller, mobile terminal or other handheld terminal. The terminal 600 may be provided with a client, such as a video player client, a browser client, a news client, an education client, etc., for playing videos.

[0092] A control device, used for sending an object recognition instruction for a target object in a video frame image corresponding to a current playback time point to the terminal 600 during the video playback process of the terminal 600;

[0093] Terminal 600 is used to respond to an object recognition instruction, obtain a video frame image corresponding to at least one target playback time point associated with a current playback time point; filter out video frame images that meet video recognition conditions as candidate video frame images from the acquired video frame images and the video frame images corresponding to the current playback time point; perform target object recognition on the candidate video frame images to obtain a recognition result, and display object information of the target object based on the recognition result.

[0094] Based on the above description of the information display system of the embodiment of the present application, the information display method provided by the embodiment of the present application is described below. Figure 2 , Figure 2 : is a flowchart of the information display method provided in the embodiment of the present application; in some embodiments, the information display method can be implemented by the terminal alone, or by the terminal and the server in collaboration. Taking the terminal alone as an example, the information display method provided in the embodiment of the present application includes:

[0095] Step 201: During video playback, the terminal receives an object recognition instruction for a target object in a video frame image corresponding to a current playback time point.

[0096] In actual implementation, the terminal presents the played video through the display screen. During the video playback, when the user wants to know the target object in the video frame image corresponding to the current playback time point, the object recognition instruction for the target object can be triggered. The current playback time point here is the time point at which the video is played when the terminal receives the object recognition instruction. For example, when the object recognition instruction is received, the video is played for 30:01, then the current playback time point is 30:01.

[0097] Here, the object recognition instruction can be triggered based on the terminal itself. For example, the terminal can present an object recognition function item for triggering the object recognition instruction. When the user triggers the object recognition function item, the terminal receives an object recognition instruction for the target object in the video frame image corresponding to the current playback time point; or, a physical button for triggering the object recognition instruction is provided on the terminal. When the user triggers the physical button, the terminal receives an object recognition instruction for the target object in the video frame image corresponding to the current playback time point; or, the terminal can also receive the user's voice content and recognize the voice content. When the voice content is used to indicate the triggering of the object recognition instruction, the terminal receives the object recognition instruction for the target object in the video frame image corresponding to the current playback time point.

[0098] In some embodiments, the object recognition instruction can also be sent to the terminal by the control device. Here, the control device is an auxiliary device used to control the terminal to play the video. For example, when the terminal is a TV terminal, the control device can be a remote control device (such as a remote control, a mobile phone terminal or other handheld terminal). The user triggers the object recognition instruction for the target object in the video frame image corresponding to the current playback time point through the remote control device, and then the remote control device sends the object recognition instruction to the terminal; when the terminal is a desktop computer, the control device can be the keyboard, mouse, etc. of the desktop computer. For example, the user can trigger the button on the keyboard for triggering the object recognition instruction to trigger the object recognition instruction for the target object in the video frame image corresponding to the current playback time point, and then the keyboard sends the object recognition instruction to the desktop computer.

[0099] In some embodiments, an object recognition instruction for a target object in a video frame image corresponding to a current playback time point may be received in the following manner: an object recognition instruction for a target object in a video frame image corresponding to a current playback time point is received from a control device; wherein the object recognition instruction is triggered based on an object recognition button on the control device.

[0100] In actual implementation, an object recognition button is provided on the control device. After receiving a trigger operation for the object recognition button, the control device sends an object recognition instruction to the terminal. The object recognition instruction here can be sent in the form of a modulated signal, that is, the control device sends a modulated signal carrying the object recognition instruction to the terminal. After receiving the modulated signal, the terminal demodulates the modulated signal to obtain the object recognition instruction received for the target object in the video frame image corresponding to the current playback time point.

[0101] In some embodiments, a touch area can also be set on the control device, and the user can perform a sliding operation in the touch area, and the terminal obtains a sliding trajectory corresponding to the sliding operation; based on the sliding trajectory, when it is determined that the sliding trajectory is used to trigger an object recognition instruction, the control device will send an object recognition instruction for the target object in the video frame image corresponding to the current playback time point to the terminal, and the terminal receives the object recognition instruction.

[0102] In some embodiments, the control device can also receive voice content input through the voice function button of the control device. For example, the user can long press the voice function button to turn on the voice collection function of the control device, so that the control device can receive the voice content input by the user; when the voice content is used to trigger the object recognition instruction, the control device will send the object recognition instruction for the target object in the video frame image corresponding to the current playback time point to the terminal, and the terminal receives the object recognition instruction.

[0103] In some embodiments, if the video frame image corresponding to the current playback time point includes only one object, then the object is the target object; if the video frame image corresponding to the current playback time point includes multiple objects, then it is necessary to determine one or more of the multiple objects as target objects, or all objects are taken as target objects, that is, the number of target objects is one or more.

[0104] In actual implementation, when it is necessary to determine one of multiple objects as the target object, the object selected by the user can be used as the target object based on the user's choice, or the terminal can select it according to corresponding selection rules, such as selecting the object with the largest face area as the target object; the terminal can also select randomly.

[0105] In some embodiments, before receiving an object recognition instruction for a target object in a video frame image corresponding to a current playback time point, the terminal may also respond to an object recognition start instruction and present a target object selection interface when the video frame image corresponding to the current playback time point contains at least two objects; and when receiving a selection operation triggered by the target object selection interface, use the object corresponding to the selection operation as the target object.

[0106] In actual implementation, the user can first trigger the object recognition start instruction. When the terminal receives the object recognition start instruction, it detects the number of objects in the video frame image corresponding to the current playback time point. When it is detected that the number of objects in the video frame image corresponding to the current playback time point is two or more, it presents a target object selection interface, so that the user can perform a target object selection operation based on the target object selection interface, and use the object corresponding to the selection operation as the target object. After determining the target object, the terminal can directly and automatically trigger the object recognition instruction, or the user can trigger the object recognition instruction.

[0107] In actual applications, the terminal can send the video frame image corresponding to the current playback time point to the server, and the server performs image recognition processing on the video frame image corresponding to the current playback time point. When it is recognized that the video frame image corresponding to the current playback time point contains an object, the object is determined as the target object; when it is recognized that the video frame image corresponding to the current playback time point contains multiple objects, a presentation instruction of the selection interface is sent to the terminal, and the terminal presents the selection interface of the target object.

[0108] In some embodiments, the terminal can present a video frame image corresponding to the current playback time point in the selection interface, and present an object selection button at the corresponding position of each object in the video frame image. When the user wants to know the information of an object, he can click the object selection button corresponding to the object, thereby triggering a selection operation. The terminal receives the selection operation and uses the object selected by the selection operation as the target object.

[0109] See also Figure 3 , Figure 3 A schematic diagram of the target object selection interface provided in an embodiment of the present application, wherein the video frame image corresponding to the current playback time point contains two objects, and a corresponding object selection button is presented next to each object. When the user clicks the object selection button 301 indicating the selection of the girl on the right, the girl on the right is selected as the target object.

[0110] In some embodiments, when the video frame image corresponding to the current playback time point contains at least two objects, the selection of the target object can be achieved through the control device, that is, the selection operation is triggered by the control device, and then the control device sends the selection instruction to the terminal, and the terminal selects the object indicated by the selection instruction as the target object.

[0111] See also Figure 4 , Figure 4 This is a schematic diagram of the target object selection interface provided by an embodiment of the present application. The video frame image corresponding to the current playback time point contains three objects. The terminal can present an object selection box at the position of one of the objects, such as the first object; the user can control the movement of the object selection box through the control device, such as the terminal can trigger the selection operation through the control device, the control device sends a modulated signal carrying corresponding selection indication information to the terminal, the terminal demodulates the modulated signal, obtains the selection instruction, and controls the selection box to move to the position of the second object; when the object selection box moves to the position of the object the user wants to select, the confirmation operation is triggered by clicking the OK button, the control device sends a modulated signal carrying the selection instruction to the terminal, the terminal receives the modulated signal, demodulates the modulated signal, and uses the object corresponding to the current object selection box as the target object.

[0112] Step 202: In response to the object recognition instruction, a video frame image corresponding to at least one target playback time point associated with the current playback time point is acquired.

[0113] Here, the time interval between the target playback time point and the current playback time point satisfies the interval condition.

[0114] In actual implementation, the terminal can send the current playback time point to the server, so that the server determines at least one target playback time point associated with the current playback time point based on the current playback time point, and then the server performs screenshot processing based on at least one target playback time point to obtain the video frame image corresponding to each target playback time; finally, the server sends the obtained video frame image to the terminal, so that the terminal obtains the video frame image corresponding to at least one target playback time point.

[0115] When the terminal obtains a complete video, the terminal may also perform screenshot processing, that is, the terminal performs screenshot processing based on at least one target playback time point to obtain a video frame image corresponding to each target playback time.

[0116] In some embodiments, when at least one target playback time point includes a first number of backward playback time points and a second number of forward playback time points, the target playback time point can be determined in the following manner: obtain a target time interval; based on the target time interval, starting from the target playback time point and moving backward, obtain a backward playback time point as the target playback time point every target time interval, until the first number of backward playback time points are obtained; based on the target time interval, starting from the target playback time point and moving forward, obtain a forward playback time point as the target playback time point every target time interval, until the second number of forward playback time points are obtained, so as to obtain at least one forward playback time point.

[0117] In actual implementation, when obtaining the target playback time point, it can be obtained at equal intervals. Here, the target time interval, the first number and the second number can be preset. For example, when the target time interval is 200ms, the first number is 7, and the second number is 2, a forward playback time point is determined every 200ms based on the current playback time point, until 7 forward playback time points are determined; then, a backward playback time point is determined every 200ms based on the current playback time point, until 2 backward playback time points are determined, so a total of 9 target playback time points are obtained.

[0118] In some embodiments, the target playback time point may be determined by: determining a playback time period including the current playback time point; and selecting at least one playback time point from the playback time period as the target playback time point.

[0119] In actual implementation, when obtaining the target playback time point, it can be randomly obtained from the playback time period including the current playback time point. For example, when the current playback time point is 30:01, a playback time period including the current playback time point can be determined, such as 30:00-30:02, then at least one playback time point is selected from 30:00-30:02 as the target playback time point. The number of target playback time points here can be pre-set. For example, when the number of target playback time points is 5, 5 playback time points are randomly selected from 30:00-30:02 as the target playback time. It should be noted that the target playback time here should be different from the current playback time point.

[0120] In actual applications, when determining the playback time period, the time intervals between the front and rear endpoints of the playback time period and the current playback time point can be the same or different. For example, when the current playback time point is 30:01, the playback time period can be 30:00-30:02 or 30:00-30:03.

[0121] Step 203: Filter out video frame images that meet the video recognition condition from the acquired video frame images and the video frame images corresponding to the current playback time point as candidate video frame images.

[0122] In actual implementation, the video recognition condition here can be that the face area size of the target object in the video frame image meets the condition, or the clarity of the target object in the video frame image meets the condition, or other video recognition conditions, which are not limited here.

[0123] In some embodiments, video frame images that meet the video recognition condition can be screened out as candidate video frame images in the following manner: determine the similarity between each acquired video frame image and the video frame image corresponding to the current playback time point; from the acquired video frame images, screen out video frame images whose similarity reaches a similarity threshold; from the video frame images whose similarity reaches the similarity threshold and the video frame images corresponding to the current playback time point, screen out video frame images that meet the video recognition condition as candidate video frame images.

[0124] Here, if the similarity between a certain video frame image and the video frame image corresponding to the current playback time point does not reach the similarity threshold, it means that the video frame image may have undergone a lens switch relative to the video frame image corresponding to the current playback time point, or the video frame image no longer contains the target object, so it has no reference value and needs to be deleted. In actual implementation, the similarity between each acquired video frame image and the video frame image corresponding to the current playback time point is determined, and then each similarity is compared with the similarity threshold to screen out the video frame images whose similarity reaches the similarity threshold. When determining the candidate video frame images, only the video frame images whose similarity reaches the similarity threshold and the video frame images corresponding to the current playback time point are screened.

[0125] In practical applications, similarity matching can be performed based on the entire video frame image, or only based on the face area in the video frame image. For example, an image of the corresponding face area is captured from each video frame image; the image of the corresponding face area captured from each video frame image and the image of the corresponding face area of ​​the video frame image corresponding to the current playback time point are similarly matched to obtain the similarity between each video frame image and the video frame image corresponding to the current playback time point.

[0126] Here, when similarity matching is performed based on the face area in the video frame image, if the similarity between the image of the corresponding face area in a certain video frame image and the image of the face area corresponding to the video frame image corresponding to the current playback time point reaches the similarity threshold, then even if the shot is switched, the video frame image still contains the target object, then the video frame image can also be used as a candidate video frame image to realize the recognition of the target object.

[0127] In some embodiments, video frame images that meet video recognition conditions can be screened out from the acquired video frame images as candidate video frame images in the following manner: determine the clarity of the target object in each acquired video frame image and the clarity of the target object in the video frame image corresponding to the current playback time point; sort the acquired video frame images and the video frame images corresponding to the current playback time point in order of clarity from large to small to obtain a first video frame image sequence; starting from the first video frame image in the first video frame image sequence, obtain a target number of video frame images as candidate video frame images.

[0128] In actual implementation, due to reasons such as blurring, the clarity of the target object in the video frame image may be low, which is not conducive to the recognition of the target object, and thus leads to a low accuracy rate in the recognition of the target object. Based on this, the terminal performs clarity detection on each acquired video frame image and the portion corresponding to the target object in the video frame image corresponding to the current playback time point, so as to obtain the clarity of the target object in each video frame image, and then selects the video frame images with the highest clarity as candidate video frame images according to the clarity. For example, when the number of targets is 2, the acquired video frame images are sorted from large to small according to the clarity to obtain the first video frame image sequence; the first video frame image and the second video frame image are obtained from the first video frame image sequence as candidate video frame images. Here, the number of targets can also be 1, then the video frame image with the highest clarity is determined as the candidate video frame image.

[0129] It should be noted that the images may be sorted in ascending order of clarity. Then, when selecting candidate video frame images, starting from the last video frame image, a target number of video frame images are obtained as candidate video frame images.

[0130] In some embodiments, a clarity threshold may also be set to filter out images whose clarity reaches the clarity threshold from the acquired video frame images as video frame images.

[0131] In some embodiments, video frame images that meet video recognition conditions can be screened out from the acquired video frame images as candidate video frame images in the following manner: determine the face area size of the target object in each acquired video frame image, and the face area size of the video frame image corresponding to the current playback time point; sort the acquired video frame images in order of face area size from large to small to obtain a second video frame image sequence; starting from the first video frame image in the second video frame image sequence, obtain a target number of video frame images as candidate video frame images.

[0132] In actual implementation, the larger the face area size, the easier it is for the corresponding target object to be identified. Based on this, the face area of ​​the target object in each video frame image is identified. Here, only the face area size needs to be identified, and there is no need to specifically identify the object information of the target object. After obtaining the face area size of each video frame image, the video frame images with the largest target face area size are selected as candidate video frame images. For example, when the target number is 2, the acquired video frame images are sorted from large to small according to the face area size to obtain a second video frame image sequence; the first video frame image and the second video frame image are obtained from the second video frame image sequence as candidate video frame images. Here, the target number can also be 1, then the video frame image with the largest face area size is determined as the candidate video frame image.

[0133] In some embodiments, video frame images that meet video recognition conditions can be screened out as candidate video frame images from the acquired video frame images in the following manner: determine the face area size of the target object in each acquired video frame image, and the face area size of the video frame image corresponding to the current playback time point; obtain a face area size threshold; and screen out video frame images whose face area size reaches the face area size threshold from the acquired video frame images and the video frame images corresponding to the current playback time point as candidate video frame images.

[0134] In actual implementation, a face area threshold can be preset, and the face area size of the target object in each video frame image and the face area size of the video frame image corresponding to the current playback time point are compared with the face area size threshold. If the face area size of a video frame image reaches the face area size threshold, the video frame image is used as a candidate video frame image.

[0135] Here, if there is no video frame image that reaches the face region size threshold, the video frame image with the largest face region size may be used as a candidate video frame image.

[0136] Step 204: performing target object recognition on the candidate video frame image to obtain a recognition result, and displaying object information of the target object based on the recognition result.

[0137] Here, the number of candidate video frame images may be one or more. When there is one candidate video frame image, the target object in the candidate video frame image is identified to obtain a recognition result. Here, the recognition result is used to indicate the target entity corresponding to the target object. Specifically, it can be used to indicate the person or object corresponding to the target object. In some embodiments, the recognition result may be identification information of the target object, such as the name of the target object. When the target entity corresponding to the target object is a celebrity, the recognition result may be the name or alias of the celebrity. When there are multiple candidate video frame images, the target object is identified for each candidate video frame image to obtain a recognition result corresponding to each candidate video frame image. Then, the recognition results of multiple candidate video frame images are combined to obtain a final recognition result. For example, the recognition result of the most candidate video frame images may be used as the recognition result of the target object. For example, there are 5 candidate video frame images in total, the recognition result of one candidate video frame image is A, and the recognition results of the other four candidate video frame images are B, then B is determined to be the recognition result of the target object.

[0138] In actual implementation, after obtaining the recognition result of the target object, the terminal obtains the object information corresponding to the recognition result and presents the object information on the display screen of the terminal, for example, it can be presented in the video screen through a floating window or through a mask with a preset transparency. Figure 5 , Figure 5 5 is a schematic diagram of presenting the object information of the target object provided by the embodiment of the present application. Here, when the target object is identified as "Fan X", the object information 501 of the target object "Fan X" is presented in the video screen through a floating window, including the object information such as name, height and weight.

[0139] In some embodiments, when there are multiple target objects, object information of multiple target objects may be presented simultaneously, or object information of one target object may be presented, and then the presented object information of the target object may be switched according to a user's selection operation.

[0140] In some embodiments, the target object can be identified in the candidate video frame image in the following manner: perform face recognition on the target image in the candidate video frame image to obtain at least two candidate recognition results corresponding to the target object and the corresponding credibility; obtain an object list of the corresponding video; based on the importance of each object in the object list in the video, weight the credibility corresponding to each candidate recognition result to obtain the weighted credibility corresponding to each candidate recognition result; and use the candidate recognition result with the highest weighted credibility as the recognition result of the target object.

[0141] Here, for videos such as movies and TV series, there is usually an object list (such as an actor list), and the target object is usually an object in the object list. Based on this, when identifying the target object, the object list can be combined to improve the accuracy of target object identification.

[0142] In actual implementation, when performing face recognition on the target object in the target image, multiple possible recognition results corresponding to the target object are usually obtained, that is, multiple candidate object information and corresponding credibility. For example, for the same face, its recognition results may be: candidate recognition result A1, credibility 90%, candidate recognition result A2, credibility 65%, candidate recognition result A3, credibility 50%; here, according to the importance of each object in the object list in the video, the weight corresponding to each object is set, so as to perform weighted processing on the credibility of the corresponding candidate recognition result according to the weight corresponding to each object. Here, the higher the importance, the greater the weight. For example, A1, A2, and A3 are all objects in the object list, the weight corresponding to A1 is 2, the weight corresponding to A2 is 1.5, and the weight corresponding to A3 is 1. Then, the credibility of A1 after weighting is 180%, the credibility of A2 after weighting is 97.5%, and the credibility of A3 after weighting is 50%. It should be noted that if a candidate recognition result is not an object in the object list, then the weight of the candidate recognition result is 0.

[0143] In some embodiments, the credibility corresponding to each candidate recognition result can be weighted based on the importance of each object in the object list in the video in the following manner: obtain the duration of each object in the object list appearing in the video; determine the weight of each object based on the duration of each object appearing in the video; and weight the credibility corresponding to the candidate object based on the weight of the object corresponding to each candidate recognition result.

[0144] In actual implementation, the importance of each object in the video can be determined based on the length of time each object appears in the video. Here, the longer the object appears in the video, the more important it is in the video, that is, the weight should be proportional to the length of time the object appears in the video.

[0145] In some embodiments, the importance of an object in a video can also be determined based on the order of arrangement of the objects in the object list. The earlier the object is in the object list, the more important the object is in the video. For example, the object list includes S1, S2, ..., S10, a total of 10 objects. Here, the weight corresponding to each object can be determined in order, such as 2.0, 1.9, ..., 1.1.

[0146] In some embodiments, the target object can be identified in the candidate video frame image in the following manner: perform face recognition on the target image in the candidate video frame image to obtain at least two candidate recognition results corresponding to the target object and the corresponding credibility; obtain an object list of the corresponding video, and match each candidate recognition result with the object in the object list; from the at least two candidate recognition results, filter out the candidate recognition results with matching objects; from the screened candidate recognition results, determine the candidate recognition result with the highest credibility as the recognition result of the target object.

[0147] In actual implementation, the target object must be an object in the object list. Based on this, the candidate object information can be screened according to the object list. If there is no object corresponding to a candidate recognition result in the object list, then the candidate recognition result must not be the recognition result of the target object. Based on this, the candidate recognition results with matching objects are screened out from at least two candidate recognition results; then, the candidate recognition result with the highest credibility is determined from the screened candidate recognition results as the recognition result of the target object.

[0148] During video playback, the present application receives an object recognition instruction for a target object in a video frame image corresponding to a current playback time point; in response to the object recognition instruction, obtains a video frame image corresponding to at least one target playback time point associated with the current playback time point; selects video frame images that meet video recognition conditions from the obtained video frame images and the video frame images corresponding to the current playback time point as candidate video frame images; performs target object recognition on the candidate video frame images to obtain recognition results, and displays object information of the target object based on the recognition results; in this way, since the present application not only obtains the video frame image corresponding to the current playback time point, but also obtains the video frame image corresponding to at least one target playback time point associated with the current playback time point, the selected candidate video frame images are more conducive to object recognition, thereby improving the accuracy of target object recognition.

[0149] Figure 6 is a flow chart of the information display method provided in the embodiment of the present application, see Figure 6 , the information display method provided in the embodiment of the present application includes:

[0150] Step 601: During video playback, the terminal receives an object recognition instruction sent by a control device.

[0151] Here, the object recognition instruction is used to instruct the recognition of the target object in the video frame image corresponding to the current playback time point. The control device is an auxiliary device used to control the terminal to play the video. For example, when the terminal is a TV terminal, the control device can be a remote control device (such as a remote control, a mobile phone terminal or other handheld terminal). The user triggers the object recognition instruction for the target object in the video frame image corresponding to the current playback time point through the remote control device, and then the remote control device sends the object recognition instruction to the terminal; when the terminal is a desktop computer, the control device can be the keyboard, mouse, etc. of the desktop computer. For example, the user can trigger the key on the keyboard for triggering the object recognition instruction to trigger the object recognition instruction for the target object in the video frame image corresponding to the current playback time point, and then the keyboard sends the object recognition instruction to the desktop computer.

[0152] As an example, when the terminal is a TV terminal and the control device is a remote control, the user triggers an object recognition operation for a target object in a video frame image corresponding to a current playback time point through the remote control. The remote control sends a modulated signal carrying an object recognition instruction. After receiving the modulated signal, the terminal demodulates the modulated signal to obtain an object recognition instruction for the target object in the video frame image corresponding to the current playback time point.

[0153] In some embodiments, an object recognition button is provided on the control device, and after receiving a trigger operation for the object recognition button, the control device sends an object recognition instruction to the terminal.

[0154] In some embodiments, a touch area can also be set on the control device, and the user can perform a sliding operation in the touch area, and the terminal obtains a sliding trajectory corresponding to the sliding operation; based on the sliding trajectory, when it is determined that the sliding trajectory is used to trigger an object recognition instruction, the control device will send an object recognition instruction for the target object in the video frame image corresponding to the current playback time point to the terminal, and the terminal receives the object recognition instruction.

[0155] In some embodiments, the control device can also receive voice content input through the voice function button of the control device. For example, the user can long press the voice function button to turn on the voice collection function of the control device, so that the control device can receive the voice content input by the user; when the voice content is used to trigger an object recognition instruction, the control device sends the object recognition instruction to the terminal, and the terminal receives the object recognition instruction.

[0156] Step 602: In response to the object recognition instruction, display object information of the target object.

[0157] The object information is obtained by performing target object recognition on video frame images corresponding to the current playback time point and video frame images corresponding to at least one target playback time point associated with the current playback time point, which meet the video recognition conditions.

[0158] In actual applications, the terminal responds to the object recognition instruction and obtains the video frame image corresponding to at least one target playback time point associated with the current playback time point; from the obtained video frame images and the video frame images corresponding to the current playback time point, the video frame images that meet the video recognition conditions are screened out as candidate video frame images; based on the candidate video frame images, the target object is recognized to obtain a recognition result; the object information corresponding to the recognition result is obtained, and the obtained object information is displayed.

[0159] By applying the above-mentioned embodiments of the present application, since the present application not only obtains the video frame image corresponding to the current playback time point, but also obtains the video frame image corresponding to at least one target playback time point associated with the current playback time point, the screened candidate video frame images are more conducive to object recognition, thereby improving the accuracy of target object recognition.

[0160] The following is a flow chart of the information display method provided by the embodiment of the present application. The information display method provided by the embodiment of the present application can be implemented by the control device, the TV terminal and the server in collaboration. Figure 7 , Figure 7 : is a flow chart of the information display method provided in the embodiment of the present application. The information display method provided in the embodiment of the present application includes:

[0161] Step 701: The TV terminal plays the video.

[0162] Step 702: The control device receives a trigger operation for an object recognition button, and in response to the trigger operation, sends a modulated signal carrying an object recognition instruction to the television terminal.

[0163] Step 703: The TV terminal sends the current playback time point to the server.

[0164] Step 704: The server obtains multiple target playback time points associated with the current playback time point.

[0165] Step 705: The server performs screenshot processing based on the current playback time point to obtain a video frame image corresponding to the current playback time point, and performs screenshot processing based on multiple target playback time points to obtain a video frame image corresponding to at least one target playback time point.

[0166] Step 706: The server obtains the similarity between each video frame image and the video frame image corresponding to the current playback time point.

[0167] Step 707: Filter out video frame images whose similarity reaches a similarity threshold from the acquired video frame images.

[0168] Step 708: Obtain the clarity of the target object in each screened video frame image and the clarity of the target object in the video frame image corresponding to the current playback time point.

[0169] Step 709: taking the video frame image with the highest definition as the candidate video frame image.

[0170] Step 710: Perform face recognition on the target object in the candidate video frame image to obtain at least two candidate recognition results corresponding to the target object and corresponding credibility.

[0171] Step 711: Get the object list corresponding to the video.

[0172] Step 712: Based on the importance of each object in the object list in the video, weight the credibility corresponding to each candidate recognition result to obtain the weighted credibility corresponding to each candidate recognition result.

[0173] Step 713: Use the candidate recognition result with the highest weighted credibility as the recognition result of the target object, search for object information corresponding to the recognition result, and send it to the TV terminal.

[0174] Step 714: Display the received object information.

[0175] By applying the above-mentioned embodiments of the present application, since the present application not only obtains the video frame image corresponding to the current playback time point, but also obtains the video frame image corresponding to at least one target playback time point associated with the current playback time point, the screened candidate video frame images are more conducive to object recognition, thereby improving the accuracy of target object recognition.

[0176] The following is a flow chart of the information display method provided by the embodiment of the present application. The information display method provided by the embodiment of the present application can be implemented by the terminal and the server in collaboration, see Figure 8 , Figure 8 : is a flow chart of the information display method provided in the embodiment of the present application. The information display method provided in the embodiment of the present application includes:

[0177] Step 801: The terminal plays the video and presents an object recognition function item.

[0178] Step 802: In response to a trigger operation on an object recognition function item, the current playback time point is sent to a server.

[0179] Step 803: The server obtains multiple target playback time points associated with the current playback time point.

[0180] Step 804: The server performs screenshot processing based on the current playback time point to obtain a video frame image corresponding to the current playback time point, and performs screenshot processing based on multiple target playback time points to obtain a video frame image corresponding to at least one target playback time point.

[0181] Step 805: Determine the similarity between each acquired video frame image and the video frame image corresponding to the current playback time point.

[0182] Step 806: Filter out video frame images whose similarity reaches a similarity threshold from the acquired video frame images.

[0183] Step 807: Obtain the face region size of the target object in each screened video frame image and the face region size of the target object in the video frame image corresponding to the current playback time point.

[0184] Step 808: taking the video frame image with the largest face region size as the candidate video frame image.

[0185] Step 809: Perform face recognition on the target object in the candidate video frame image to obtain at least two candidate recognition results corresponding to the target object and corresponding credibility.

[0186] Step 810: Obtain an object list corresponding to the video, and match each candidate recognition result with an object in the object list.

[0187] Step 811: Filter out candidate recognition results with matching objects from at least two pieces of candidate object information.

[0188] Step 812: From the candidate object information obtained by screening, determine the candidate recognition result with the highest credibility as the recognition result of the target object, search for the object information corresponding to the recognition result, and send it to the terminal.

[0189] Step 813: Display the received object information.

[0190] By applying the above-mentioned embodiments of the present application, since the present application not only obtains the video frame image corresponding to the current playback time point, but also obtains the video frame image corresponding to at least one target playback time point associated with the current playback time point, the screened candidate video frame images are more conducive to object recognition, thereby improving the accuracy of target object recognition.

[0191] The following will describe an exemplary application of the embodiment of the present application in an actual application scenario. Taking a terminal as a television terminal as an example, the information display method provided by the embodiment of the present application is described. Fig. 9 is a flow chart of the information display method provided in the embodiment of the present application, see Fig. 9, the information display method provided in the embodiment of the present application includes:

[0192] Step 901: The video client plays the video.

[0193] Step 902: The remote control device sends an object recognition instruction to the video client.

[0194] In actual implementation, when a TV terminal plays a video and a user sees a face while watching the video, the object recognition function (AI magic mirror function) can be triggered by pressing the object recognition button on the remote control device, and the remote control device sends the object recognition command to the TV terminal.

[0195] Here, the object recognition instruction is used to instruct to recognize the target object in the video frame image corresponding to the current playback time point, wherein the number of the target objects can be one or more.

[0196] Step 903: The video client sends the current play time point and video information to the TV backend.

[0197] Here, the current playback time point refers to the playback time point to which the video is played when the object recognition instruction is received.

[0198] Step 904: the TV background sends a request to obtain the video frame image to the screenshot background.

[0199] Here, the request for obtaining the video frame image carries the current playback time point and film information.

[0200] Step 905: Perform screenshot processing based on the object playback time point and the film information to obtain a candidate video frame image.

[0201] For image recognition, the quality of the incoming image is directly related to the recognition rate. The unit of the current playback time point sent to the screenshot backend is milliseconds. Due to the change of the video screen, it is very likely that the screen has changed when the user sees the screen and the remote control device triggers the recognition function. In order to make up for this deficiency, the server is used to capture multiple pictures for improvement. In actual implementation, Fig.10 is a flowchart of the process of obtaining the candidate video frame image provided by the embodiment of the present application, see Fig.10 , the process of obtaining the candidate video frame image provided in the embodiment of the present application includes:

[0202] Step 1001: Determine the current playback time point T1.

[0203] Step 1002: Perform screenshot processing based on T1 to obtain a video frame image P1 corresponding to T1.

[0204] Step 1003: Generate target playback time points T01-T09 according to T1.

[0205] Here, taking T1 as the reference time, a target playback time point is determined every 200ms, and a total of 7 target playback time points are determined, namely T01-T07; a screenshot is taken every 200ms, and a total of 7 screenshots are taken. Figure 2 Zhang, that is, T08-T09.

[0206] Step 1004: Perform screenshot processing based on T01-T09 to obtain video frame images P01-P09 corresponding to T01-T09.

[0207] Step 1005: Delete the video frame images among P01-P09 whose similarity with P1 does not reach the similarity threshold.

[0208] Here, P101-P109 are compared with the reference image P1, and video frame images with extremely low similarity to P1 (video frame images whose similarity does not reach the similarity threshold) are deleted. Low similarity means that the screen may have undergone a lens switch and has no reference value.

[0209] Step 1006: Filter out the video frame image with the largest face region size from the remaining video frame images as the candidate video frame image.

[0210] The remaining video frame images are subjected to face region recognition in turn. No specific object recognition is required. Only the face region needs to be analyzed, and the image P with the largest face region size is taken as the candidate video frame image.

[0211] Step 906: The screenshot backend sends the obtained Uniform Resource Locator (URL) of the candidate video frame image to the TV backend.

[0212] Step 907: the TV backend sends the URL of the candidate video frame image to the video client.

[0213] Step 908: The video client sends an object recognition request carrying the URL of the candidate video frame image to the TV backend.

[0214] Step 909: The TV backend sends an object recognition request to the recognition backend.

[0215] Step 910: The recognition background recognizes the target object based on the candidate video frame images and the object list of the video.

[0216] In actual implementation, each video is composed of an object list (actor list), and the relationship between the video and the object list is as follows:

[0217] One VID (video id) corresponds to multiple CIDs (album ids);

[0218] One CID (album id) corresponds to multiple VIDs (video ids);

[0219] One CID (album id) corresponds to multiple StarIDs (object ids);

[0220] It can be seen that as long as the CID is obtained, the object list of the video can be obtained. In actual applications, the object recognition request carries the URL and CID of the candidate video frame image, and the recognition background can obtain the object list based on the CID.

[0221] Fig.12 is a flowchart of the object recognition process provided by the embodiment of the present application, see Fig.12 , the object recognition process provided in the embodiment of the present application includes:

[0222] Step 1201: Obtain the URL and CID of the candidate video frame image.

[0223] Step 1202: Obtain an object list according to the CID.

[0224] Step 1203: Initialize the weights of the objects in the object list.

[0225] According to the object list, weights are assigned to each object in order from the main role to the supporting role; for example, if there are S1, S2, ..., S10 objects in the object list, their corresponding weighted values ​​are 2.0, 1.9, ..., 1.1.

[0226] Step 1204: Perform face recognition on the candidate video frame image according to the URL of the candidate video frame image.

[0227] Here, face recognition is performed on the candidate video frame images through a face recognition interface, which extracts and analyzes facial features in images and video sources based on deep learning algorithms and massive data sets. Fig.11 This is an example diagram of the face recognition interface provided by the embodiment of the present application, see Fig.11 The face recognition interface can realize accurate face detection and face recognition, covering face detection and analysis, facial features positioning, face comparison and search, cross-age face recognition, etc., providing strong technical support for application scenarios such as face beautification and intelligent album classification.

[0228] For example, for the same candidate video frame image, the candidate recognition results are: recognition result A1, with a credibility of 90%, recognition result A2, with a credibility of 65%, and recognition result A3, with a credibility of 50%.

[0229] Step 1205: Obtain the candidate recognition results and corresponding credibility.

[0230] Step 1206: Associating the credibility with the weighted value.

[0231] In actual implementation, it is determined whether A1, A2, and A3 are consistent with the objects in S1-S10. If they are consistent, the corresponding credibility is multiplied by the weighted value to obtain the final credibility.

[0232] Step 1207: Determine the candidate recognition result with the highest weighted credibility as the recognition result of the target object.

[0233] Step 911: The identification background sends the identification result to the TV background.

[0234] Step 912: The TV backend obtains the object information of the target object according to the recognition result.

[0235] In actual implementation, the TV background searches for the target object's object information, such as height, birthday, etc., based on the recognition results.

[0236] Step 913: The TV backend returns the object information of the target object to the video client.

[0237] Step 914: Display object information of the target object.

[0238] Here, when there are multiple target objects, the object information of multiple target objects can be displayed at the same time, or the object information of one target object can be displayed, and then the object information of the displayed target object can be switched based on the user's switching operation. Fig.13 This is a schematic diagram of the interface for displaying object information provided in the embodiment of the present application, see Fig.13 The video frame image corresponding to the current playback time point includes two target objects, and the object information 1301 of one of the target objects is presented, including name, height, birthday, etc.; at the same time, the name of the other target object is presented to indicate the switching of the object information. When the switching operation is received, the object information of the other target object is presented.

[0239] By applying the above-mentioned embodiments of the present application, the recognition accuracy and success rate of the target object can be improved, thereby improving the accuracy of the object information display.

[0240] The following is a description of the information display device provided by the embodiment of the present application. Fig.14 , Fig.14 : is a structural diagram of an information display device provided in an embodiment of the present application. The video playback device provided in an embodiment of the present application includes:

[0241] The first receiving module 410 is used to receive an object recognition instruction for a target object in a video frame image corresponding to a current playback time point during video playback;

[0242] The acquisition module 420 is used to acquire a video frame image corresponding to at least one target playback time point associated with the current playback time point in response to the object recognition instruction;

[0243] A screening module 430 is used to screen out video frame images that meet the video recognition condition from the acquired video frame images and the video frame images corresponding to the current playback time point as candidate video frame images;

[0244] The first display module 440 is used to perform target object recognition on the candidate video frame image, obtain a recognition result, and display object information of the target object based on the recognition result.

[0245] In some embodiments, the first receiving module 410 is further configured to receive an object recognition instruction sent by a control device for a target object in a video frame image corresponding to a current playback time point;

[0246] The object recognition instruction is triggered based on an object recognition button on the control device.

[0247] In some embodiments, the first receiving module 410 is further configured to respond to an object recognition start instruction and present a target object selection interface when the video frame image corresponding to the current playback time point contains at least two objects;

[0248] A selection operation triggered by a selection interface based on the target object is received, and the object corresponding to the selection operation is used as the target object.

[0249] In some embodiments, when the at least one target playback time point includes a first number of backward playback time points and a second number of forward playback time points, the first receiving module 410 is further used to obtain a target time interval;

[0250] Based on the target time interval, starting from the target playback time point and moving backwards, every target time interval, obtaining a backward playback time point as a target playback point until a first number of backward playback time points are obtained;

[0251] Based on the target time interval, starting from the target playback time point and moving forward, a forward playback time point is acquired as the target playback time point every target time interval until a second number of forward playback time points are acquired.

[0252] In some embodiments, the acquisition module 420 is further used to determine a playback time period including the current playback time point;

[0253] At least one playback time point is selected from the playback time period as a target playback time point.

[0254] In some embodiments, the acquisition module 420 is further used to determine the similarity between each acquired video frame image and the video frame image corresponding to the current playback time point;

[0255] Screening out video frame images whose similarity reaches a similarity threshold from the acquired video frame images;

[0256] From the video frame images whose similarity reaches the similarity threshold and the video frame images corresponding to the current playback time point, the video frame images that meet the video recognition condition are screened out as candidate video frame images.

[0257] In some embodiments, the screening module 430 is further used to determine the clarity of the target object in each acquired video frame image and the clarity of the target object in the video frame image corresponding to the current playback time point;

[0258] The acquired video frame images and the video frame images corresponding to the current playback time point are sorted in descending order of face clarity to obtain a first video frame image sequence;

[0259] Starting from the first video frame image of the first video frame image sequence, a target number of video frame images are acquired as candidate video frame images.

[0260] In some embodiments, the screening module 430 is further used to determine the face size of the target object in each acquired video frame image and the face area size of the video frame image corresponding to the current playback time point;

[0261] The acquired video frame images and the video frame images corresponding to the current playback time point are sorted in descending order of face region size to obtain a second video frame image sequence;

[0262] Starting from the first video frame image of the second video frame image sequence, a target number of video frame images are acquired as candidate video frame images.

[0263] In some embodiments, the screening module 430 is further used to determine the size of the face region of the target object in each video frame image obtained, and the size of the face region of the video frame image corresponding to the current playback time point;

[0264] Get the face area size threshold;

[0265] From the acquired video frame images and the video frame images corresponding to the current playback time point, video frame images whose face region sizes reach the face region size threshold are screened out as candidate video frame images.

[0266] In some embodiments, the first display module 440 is further used to perform face recognition on the target image in the candidate video frame image to obtain at least two candidate recognition results corresponding to the target object and corresponding credibility;

[0267] Get a list of objects corresponding to the video;

[0268] Based on the importance of each object in the object list in the video, weighting the credibility corresponding to each candidate recognition result to obtain a weighted credibility corresponding to each candidate recognition result;

[0269] The candidate recognition result with the highest weighted credibility is taken as the recognition result of the target object.

[0270] In some embodiments, the first display module 440 is further used to obtain the duration of each object in the object list appearing in the video;

[0271] Determine the weight of each object according to the duration of each object appearing in the video.

[0272] Based on the weight of the object corresponding to each candidate recognition result, the credibility corresponding to the candidate object is weighted.

[0273] In some embodiments, the first display module 440 is further used to perform face recognition on the target image in the candidate video frame image to obtain at least two candidate recognition results corresponding to the target object and corresponding credibility;

[0274] Obtaining an object list corresponding to the video, and matching each candidate recognition result with an object in the object list;

[0275] Filtering candidate recognition results of objects that match each other from at least two pieces of candidate object information;

[0276] From the screened candidate recognition results, determine the candidate recognition result with the highest credibility as the recognition result of the target object.

[0277] The present application provides an information display device, including:

[0278] A second receiving module is used to receive an object recognition instruction sent by a control device during video playback, wherein the object recognition instruction is used to instruct to recognize a target object in a video frame image corresponding to a current playback time point;

[0279] A second display module, configured to display object information of the target object in response to the object recognition instruction;

[0280] The object information is obtained by performing target object recognition on video frame images corresponding to the current playback time point and video frame images corresponding to at least one target playback time point associated with the current playback time point, which meet the video recognition conditions.

[0281] The present application also provides a computer device, which may be a terminal or a server. Fig.15 , Fig.15 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. The computer device provided in an embodiment of the present application includes:

[0282] Memory 550, used to store executable instructions;

[0283] The processor 510 is used to implement the information display method provided in the embodiment of the present application when executing the executable instructions stored in the memory.

[0284] Here, the processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0285] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 510.

[0286] The memory 550 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 550 described in the embodiments of the present application is intended to include any suitable type of memory.

[0287] In some embodiments, at least one network interface 520 and a user interface 530 may also be included. The various components in the computer device 500 are coupled together via a bus system 540. It is understood that the bus system 540 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 540 is not described in detail. Fig.15 Various buses are labeled as bus system 540 .

[0288] The embodiment of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the information display method described in the embodiment of the present application.

[0289] The present application embodiment provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will be caused to execute the method provided by the present application embodiment, for example, Figure 2 The method shown.

[0290] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EP ROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0291] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0292] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0293] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0294] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. An information display method, characterized in that: The method comprises: During video playback, an object recognition instruction is received for a target object in a video frame image corresponding to a current playback time point; In response to the object recognition instruction, acquiring a video frame image corresponding to at least one target playback time point associated with the current playback time point; Filtering out video frame images that meet the video recognition condition from the acquired video frame images and the video frame images corresponding to the current playback time point as candidate video frame images; Performing face recognition on the target object in the candidate video frame image to obtain at least two candidate recognition results corresponding to the target object and corresponding credibility; Get a list of objects corresponding to the video; Based on the importance of each object in the object list in the video, weighting the credibility corresponding to each candidate recognition result to obtain a weighted credibility corresponding to each candidate recognition result; The candidate recognition result with the highest weighted credibility is used as the recognition result of the target object, and based on the recognition result, the object information of the target object is displayed.

2. The method according to claim 1, characterized in that The receiving of an object recognition instruction for a target object in a video frame image corresponding to a current playback time point includes: receiving an object recognition instruction for a target object in a video frame image corresponding to a current playback time point sent by a control device; The object recognition instruction is triggered based on an object recognition button on the control device.

3. The method according to claim 1, characterized in that Before receiving an object recognition instruction for a target object in a video frame image corresponding to a current playback time point, the method further includes: In response to the object recognition start instruction, when the video frame image corresponding to the current playback time point contains at least two objects, presenting a target object selection interface; A selection operation triggered by a selection interface based on the target object is received, and the object corresponding to the selection operation is used as the target object.

4. The method according to claim 1, characterized in that When the at least one target playback time point includes a first number of backward playback time points and a second number of forward playback time points, before acquiring the video frame image corresponding to the at least one target playback time point associated with the current playback time point, the method further includes: Get the target time interval; Based on the target time interval, starting from the target playback time point and moving backwards, every target time interval, obtaining a backward playback time point as the target playback time point until a first number of backward playback time points are obtained; Based on the target time interval, starting from the target playback time point and moving forward, a forward playback time point is acquired as the target playback time point every target time interval, until a second number of forward playback time points are acquired.

5. The method according to claim 1, characterized in that Before acquiring the video frame image corresponding to at least one target playback time point associated with the current playback time point, the method further includes: Determine a playback time period including the current playback time point; At least one playback time point is selected from the playback time period as a target playback time point.

6. The method according to claim 1, characterized in that The step of selecting, from the acquired video frame images and the video frame images corresponding to the current playback time point, video frame images that meet the video recognition condition as candidate video frame images includes: Determine the similarity between each acquired video frame image and the video frame image corresponding to the current playback time point; Screening out video frame images whose similarity reaches a similarity threshold from the acquired video frame images; From the video frame images whose similarity reaches the similarity threshold and the video frame images corresponding to the current playback time point, the video frame images that meet the video recognition condition are screened out as candidate video frame images.

7. The method according to claim 1, characterized in that The step of selecting, from the acquired video frame images and the video frame images corresponding to the current playback time point, video frame images that meet the video recognition condition as candidate video frame images includes: Determine the clarity of the target object in each acquired video frame image and the clarity of the target object in the video frame image corresponding to the current playback time point; Sorting the acquired video frame images and the video frame images corresponding to the current playback time point in the order of the clarity from large to small to obtain a first video frame image sequence; Starting from the first video frame image of the first video frame image sequence, a target number of video frame images are acquired as candidate video frame images.

8. The method according to claim 1, characterized in that The step of selecting, from the acquired video frame images and the video frame images corresponding to the current playback time point, video frame images that meet the video recognition condition as candidate video frame images includes: Determine the face area size of the target object in each acquired video frame image and the face area size of the video frame image corresponding to the current playback time point; The acquired video frame images and the video frame images corresponding to the current playback time point are sorted in descending order of face region size to obtain a second video frame image sequence; Starting from the first video frame image of the second video frame image sequence, a target number of video frame images are acquired as candidate video frame images.

9. The method according to claim 1, characterized in that The step of selecting, from the acquired video frame images and the video frame images corresponding to the current playback time point, video frame images that meet the video recognition condition as candidate video frame images includes: Determine the face area size of the target object in each acquired video frame image and the face area size of the video frame image corresponding to the current playback time point; Get the face area size threshold; From the acquired video frame images and the video frame images corresponding to the current playback time point, video frame images whose face region sizes reach the face region size threshold are screened out as candidate video frame images.

10. An information display method, characterized in that: The method comprises: During video playback, an object recognition instruction sent by a control device is received, wherein the object recognition instruction is used to instruct to recognize a target object in a video frame image corresponding to a current playback time point; wherein, recognizing a target object in a video frame image corresponding to the current playback time point comprises: selecting a video frame image satisfying a video recognition condition from among the acquired video frame images corresponding to at least one target playback time point associated with the current playback time point and the video frame image corresponding to the current playback time point as a candidate video frame image; performing face recognition on the target object in the candidate video frame image to obtain at least two candidate recognition results and corresponding credibility of the corresponding target object; obtaining an object list corresponding to the video; performing weighted processing on the credibility corresponding to each candidate recognition result based on the importance of each object in the object list in the video to obtain a weighted credibility corresponding to each candidate recognition result; and using the candidate recognition result with the highest weighted credibility as the recognition result of the target object; In response to the object recognition instruction, object information of the target object is displayed.

11. An information display device, characterized in that: The device comprises: A first receiving module is used to receive an object recognition instruction for a target object in a video frame image corresponding to a current playback time point during video playback; An acquisition module, configured to acquire, in response to the object recognition instruction, a video frame image corresponding to at least one target playback time point associated with the current playback time point; A screening module is used to screen out video frame images that meet the video recognition conditions from the acquired video frame images and the video frame images corresponding to the current playback time point as candidate video frame images; The first display module is used to perform face recognition on the target object in the candidate video frame image to obtain at least two candidate recognition results corresponding to the target object and the corresponding credibility; obtain an object list corresponding to the video; based on the importance of each object in the object list in the video, weight the credibility corresponding to each candidate recognition result to obtain the weighted credibility corresponding to each candidate recognition result; use the candidate recognition result with the highest weighted credibility as the recognition result of the target object to obtain a recognition result, and based on the recognition result, display the object information of the target object.

12. A computer device, characterized in that: include: A memory for storing executable instructions; A processor, configured to implement the information display method according to any one of claims 1 to 10 when executing the executable instructions stored in the memory.

13. A computer-readable storage medium, characterized in that: Executable instructions are stored, and when executed by a processor, the information display method according to any one of claims 1 to 10 is implemented.

14. A computer program product comprising executable instructions or a computer program, characterized in that When the executable instructions or computer program are executed by a processor, the information display method according to any one of claims 1 to 10 is implemented.

Images