Image searching method, video playing device and storage medium

Abstract

The invention discloses an image searching method, video playing equipment and a storage medium, and the image searching method comprises the steps: the video playing equipment freezes a video frame currently displayed by the video playing equipment in response to a received freezing instruction for a currently played video; decoding at least a part of code streams about the video in the cache of the video playing equipment to obtain a plurality of candidate video frames which are not displayed; in response to a received image searching instruction, determining a target video frame from the frozen video frame and the plurality of candidate video frames; and performing image search processing by using the target video frame. According to the scheme, the image searching accuracy can be improved.

Description

Technical Field

[0001] The present application relates to the field of image acquisition technology, and in particular to an image search method, a video playback device, and a storage medium. Background Art

[0002] With the popularization of artificial intelligence applications, image search has become possible. Taking an implementation method of image search as an example, you can input a target picture, and in a large number of videos, you can quickly locate video clips similar to the target and present them in different sortings. However, the current image search still stays at the level of searching for related images in the database given an image, which often requires pausing the video and taking a screenshot while watching the video, and then entering the screenshot into the image search application, which is more cumbersome. Summary of the invention

[0003] The present application at least provides a picture search method, a video playback device and a storage medium.

[0004] The present application provides an image search method, comprising: in response to receiving a freeze instruction for a video being played, a video playback device freezes a video frame currently displayed by the video playback device; decoding at least a portion of a code stream related to a video in a cache of the video playback device to obtain a number of candidate video frames that are not displayed; in response to receiving the image search instruction, determining a target video frame from the frozen video frame and a number of candidate video frames; and performing image search processing using the target video frame.

[0005] The present application provides an image search device, comprising: a freezing unit, a video decoding unit, a video analysis unit and an image search unit; the freezing unit is used for the video playback device to freeze the video frame currently displayed by the video playback device in response to receiving a freeze instruction for the video being played; the video decoding unit is used to decode at least part of the code stream about the video in the cache of the video playback device to obtain a number of candidate video frames that are not displayed; the video analysis unit is used to determine a target video frame from the frozen video frame and a number of candidate video frames in response to receiving the image search instruction; the image search unit is used to use the target video frame to perform image search processing.

[0006] The present application provides a video playback device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-mentioned image search method.

[0007] The present application provides a computer-readable storage medium on which program instructions are stored. When the program instructions are executed by a processor, the above-mentioned image search method is implemented.

[0008] In the above scheme, if the video playback device receives a freeze command for the video during the video playback process, it will first freeze the current display screen, and then decode the code stream in the cache to obtain several candidate video frames. When a search image command is received, the target video frame for image search can be determined from multiple video frames, which simplifies the image search process. Compared with directly using the frozen video frame for image search, this scheme determines the target video frame from multiple video frames for image search processing. For example, a video frame with better quality can be selected from multiple video frames for image search processing, which can improve the accuracy of image search.

[0009] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0011] Figure 1 It is a flowchart of an embodiment of the image search method of the present application; Figure 2 yes Figure 1 Schematic diagram of the sub-process of step S12; Figure 3 This is another flowchart of an embodiment of the image search method of the present application; Figure 4 yes Figure 1 Schematic diagram of the sub-process of step S14; Figure 5 It is a structural schematic diagram of an embodiment of the image search device of the present application; Figure 6 It is a structural schematic diagram of an embodiment of a video playback device of the present application; Figure 7 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0012] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0013] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0014] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0015] In some application scenarios, the image search method provided by the present application can be applied to video playback, etc. The execution subject for implementing the image search method described in the present application can be an image search device, a video playback device, etc. For example, the image search device can be set in a terminal device such as a video player or a server or other processing device, wherein the terminal device can be a video playback device, a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementations, the image search method can be implemented by a processor calling a computer-readable instruction stored in a memory.

[0016] See also Figure 1 , Figure 1 1 is a flow chart of an embodiment of the image search method of the present application. Figure 1 As shown, the image search method provided by the embodiment of the present disclosure may include the following steps: Step S11: in response to receiving a freeze instruction for a video being played, the video playback device freezes a video frame currently displayed by the video playback device.

[0017] The video playback device can be any device with a video playback function. The video being played proposed in this embodiment can be a real-time video or a playback video. Real-time video can be a process of capturing an ongoing event through a camera and playing it on the network with almost no delay. The playback video can be a video content that has been recorded after the fact, which can be a recording of a live video or a separately recorded video file. The freeze command can be a pause playback command for the video or a freeze button is set on the video playback interface of the video playback device. If the freeze button receives a click operation, it is determined that a freeze command for the video being played is received, or it can also be a receiving input voice, and by recognizing the collected voice, it is determined whether a freeze command is received.

[0018] Step S12: decoding at least a portion of the code stream of the video in the buffer of the video playback device to obtain a number of candidate video frames that are not displayed.

[0019] Taking into account that the image may be blurred at the moment of freezing, for example, the target object moves too fast in live broadcast and other scenarios, so that the outline of the target object is not clear, or the outline of the target object in the frozen video screen may not be fully displayed, at least part of the bitstream about the video in the cache of the video playback device may be decoded to obtain several candidate video frames that are not displayed. The cache of the video playback device may be a cache queue in the entire link required for the video playback device from video decoding to video playback. For example, the cache may include the following circular cache area. The candidate video frames obtained by decoding each bitstream in the cache have not been displayed during the video playback process. In some application scenarios, the candidate video frames have not been displayed on the display interface of the video player, so the decoded candidate video frames can exist as YUV three-channel information, that is, decoding the bitstream in the cache can obtain the YUV information of the candidate video frames.

[0020] Step S13: In response to receiving the image search instruction, determining a target video frame from the frozen video frame and a plurality of candidate video frames.

[0021] In some application scenarios, after receiving the image search instruction, the decoding of at least part of the code stream of the video in the cache of the video playback device can be stopped to improve the efficiency of the image search. In other application scenarios, after receiving the image search instruction, the decoding of at least part of the code stream of the video in the cache of the video playback device can be continued for a preset time, and the video frame can be decoded for the selection of subsequent target video frames. The triggering of the image search instruction can be to display the image search button on the display interface of the video playback device. If a click operation on the image search button is received, it is determined that the image search instruction has been received. Or an operation such as a long press on the display interface is detected, a display box containing the image search button pops up, and a click operation on the image search button is detected, then it is determined that the image search instruction has been received. The method of determining the target video frame from the frozen video frame and several candidate video frames can be to select one or more frames as the target video frame from the frozen video frame and several candidate video frames according to a pre-set selection rule, that is, the number of target video frames can be one or more.

[0022] Step S14: Perform image search processing using the target video frame.

[0023] Methods of using the target video frame for image search processing include but are not limited to the following two: one is that the target video frame can be used as the input of the image search, and the image or video clip related to the target video frame can be determined from the built-in database of the video playback device as the output to obtain the image search result. For example, the video frame related to the target video frame can be searched from the video being played as the output. For example, the target video frame can be a video frame about a certain leading actor, and the video frame with the leading actor can be selected from the video as the output of the image search to form a collection of the leading actor's video frames. The second is that the video playback device can establish a network connection with the cloud or other devices, and the video playback device can realize online image search by calling the database set in the cloud. There are many image search methods at present, and it is impossible to give examples of all of them. This solution does not limit the specific means of obtaining the image search result after obtaining the image search input.

[0024] In the above scheme, if the video playback device receives a freeze command for the video during the video playback process, it will first freeze the current display screen, and then decode the code stream in the cache to obtain several candidate video frames. When a search image command is received, the target video frame for image search can be determined from multiple video frames, which simplifies the image search process and compared with directly using the frozen video frame for image search, this scheme determines the target video frame from multiple video frames for image search processing, which can improve the accuracy of image search.

[0025] In some embodiments, the image search method may further include the following steps: obtaining a decoding capacity margin of a video playback device. In response to the decoding capacity margin satisfying a full-speed decoding condition, controlling the decoding speed of the video playback device to switch from normal decoding to full-speed decoding. The number of video frames obtained by full-speed decoding per unit time is greater than the number of video frames obtained by normal decoding.

[0026] Exemplarily, if the decoding capacity margin of the video playback device is obtained to be M 1080P@30fps, that is, the video playback device can also process M channels of 1080P resolution and 30 frames per second (fps) video streams under the current use. In some application scenarios, in response to determining that the decoding capacity margin is greater than or equal to the preset full-speed decoding margin, it can be determined that the decoding capacity margin meets the full-speed decoding condition. The method of controlling the decoding speed of the video playback device from normal decoding to full-speed decoding can be that if normal decoding needs to reserve a certain decoding capacity margin to cope with emergencies, the reserved decoding capacity margin can be reduced to achieve the effect of switching to full-speed decoding, thereby increasing the decoding speed of the video playback device. In some application scenarios, during normal decoding, a period of time is required between the decoding of the previous and next two frames of the code stream, and the interval time can be shortened to achieve the effect of switching to full-speed decoding.

[0027] In some embodiments, see Figure 2, the above step S12 may include the following steps: For each video bitstream in the buffer, perform the following steps: Step S121: Determine the decoding virtual timestamp of the bitstream to be decoded in the buffer.

[0028] The decoding virtual timestamp is obtained from the decoding virtual timestamp of the previous decoded bitstream of the bitstream to be decoded and the estimated decoding time consumption of the bitstream to be decoded. It can be understood that a candidate video frame that has not been displayed can be obtained by decoding a bitstream to be decoded in the buffer of the video playback device. Among them, the bitstreams to be decoded in the buffer are arranged in a certain order, and the one at the head of the queue can be considered as the bitstream to be decoded. The estimated decoding time consumption of the bitstream to be decoded can be the average decoding time consumption t1 of the bitstream. Among them, the average decoding time consumption t1 for decoding one frame of the bitstream is related to the width, height, frame rate of the bitstream, and the decoding capacity margin of the video playback device. In some application scenarios, in the case of full-speed decoding, the average time consumption t1 = 1000 / (M * 30 * 1920 * 1080 / W * H * fps), and in the case of normal decoding, the average time consumption t1 = 1000 / (N * 30 * 1920 * 1080 / W * H * fps), N < M, where M represents the decoding capacity margin of the video playback device. W represents the width of the bitstream, H represents the height of the bitstream, fps represents the frame rate of the bitstream, and 1000 represents 1000 ms. The width and height of the bitstream can be obtained by parsing the bitstream. The previous decoded bitstream of the bitstream to be decoded refers to the adjacent decoded bitstream in the buffer to the bitstream to be decoded. For example, since the bitstreams in the buffer are decoded in order, the latest decoded bitstream is the previous decoded bitstream of the bitstream to be decoded. The decoding virtual timestamp of the bitstream to be decoded can be represented by Tdts. Tdts = Tdv + 1000 / (M * 30 * 1920 * 1080 / W * H * fps), that is, Tdts = Tdv + t1, where Tdv represents the decoding virtual timestamp of the previous decoded bitstream.

[0029] Step S122: In response to the relationship between the decoding virtual timestamp and the system timestamp of the video playback device satisfying a preset decoding condition, determine to decode the bitstream to be decoded to obtain a candidate video frame that has not been displayed.

[0030] In some application scenarios, in response to the decoded virtual timestamp being less than the system timestamp of the video playback device, it is determined that the relationship between the two meets the preset decoding conditions. Among them, the system timestamp of the video playback device can be considered as the timestamp provided by the operating system of the video playback device. In other application scenarios, in response to the decoded virtual timestamp being less than the system timestamp of the video playback device, more stringent conditions can be set under the premise, such as the difference between the two timestamps must be greater than a certain threshold value. In some application scenarios, if the relationship between the decoded virtual timestamp and the system timestamp of the video playback device does not meet the preset decoding conditions, the code stream to be decoded is not decoded. Specifically, the code stream to be decoded can be kept at the head of the queue in the cache, and it is continuously determined whether the relationship between the decoded virtual timestamp and the system timestamp of the video playback device meets the preset decoding conditions. In response to the relationship between the decoded virtual timestamp and the system timestamp of the video playback device meeting the preset decoding conditions, it is determined that the code stream to be decoded is decoded to obtain the candidate video frame that is not displayed.

[0031] In some embodiments, the image search method may include the following steps: sequentially adding each candidate video frame to a candidate video frame queue; analyzing and processing each candidate video frame in the candidate video frame queue to obtain relevant information about the target object in each candidate video frame. On this basis, the above step S13 may include the following steps: determining the target video frame based on the relevant information about the target object in each candidate video frame and the relevant information about the target object in the frozen video frame.

[0032] As described above, inside the video playback device, each candidate video frame obtained by decoding may be information under three YUV channels of each candidate video frame, and then the candidate video frame queue may be the YUV three-channel information of each candidate video frame arranged in sequence.

[0033] The method of analyzing and processing each candidate video frame in the candidate video frame queue may be to perform target detection or target segmentation and other processing, so as to obtain relevant information about the target object in each candidate video frame. The relevant information about the target object may include the position of the target object in the video frame or the pixel information of the area where the target object is located in the video frame, such as the information of the three YUV channels of each pixel point in the area where the target object is located.

[0034] The above-mentioned method of determining the target video frame based on the relevant information about the target object in each candidate video frame and the relevant information about the target object in the frozen video frame can be to score the frozen video frame and each candidate video frame based on the pixel value of each pixel in the area where the target object is located in the frozen video frame and the pixel value of each pixel in the area where the target object is located in each candidate video frame, and determine the target video frame according to the score.

[0035] In some embodiments, the image search method may further include the following steps: removing the candidate image frames that have been analyzed and processed from the candidate video frame queue. That is, it is ensured that the candidate video frames in the candidate video frame queue are candidate image frames that have not been analyzed and processed. In some application scenarios, the candidate image frames that are being analyzed and processed can also be removed from the candidate video frame queue. On this basis, the above-mentioned response to the relationship between the decoded virtual timestamp and the system timestamp of the video playback device satisfying the preset decoding conditions, determining to decode the code stream to be decoded, and obtaining the candidate video frames that are not displayed can be specifically: in response to the relationship between the decoded virtual timestamp and the system timestamp of the video playback device satisfying the preset decoding conditions, and the number of candidate video frames contained in the candidate video frame queue does not reach the maximum capacity of the candidate video frame queue, determining to decode the code stream to be decoded, and obtaining the candidate video frames that are not displayed. The maximum capacity of the candidate video frame queue can be the maximum number of candidate video frames that the candidate video frame queue can accommodate.

[0036] Exemplarily, in response to Ts>Tdts and the number of candidate video frames included in the candidate video frame queue not reaching the maximum capacity of the candidate video frame queue, it is determined to decode the code stream to be decoded to obtain the candidate video frames that are not displayed. In response to the relationship between the decoded virtual timestamp and the system timestamp of the video playback device not satisfying the preset decoding condition, or the number of candidate video frames included in the candidate video frame queue reaching the maximum capacity of the candidate video frame queue, the step of decoding the code stream to be decoded to obtain the candidate video frames that are not displayed is not performed, and the judgment is continued or after a period of time, it is judged whether the relationship between the decoded virtual timestamp and the system timestamp of the video playback device satisfies the preset decoding condition, and whether the number of candidate video frames included in the candidate video frame queue reaches the maximum capacity of the candidate video frame queue.

[0037] The above-mentioned method of analyzing and processing each candidate video frame in the candidate video frame queue to obtain relevant information about the target object in each candidate video frame may be: performing the following steps on each candidate video frame in the candidate video frame queue: Figure 3 The following steps are shown: Step S21: Determine the analysis virtual timestamp of the candidate video frame to be analyzed in the candidate video frame queue.

[0038] The candidate video frames in the candidate video frame queue can be analyzed and processed in sequence, that is, the candidate video frames in the candidate video frame queue can be used as candidate video frames to be analyzed in sequence. Optionally, the analysis virtual timestamp is obtained by the analysis virtual timestamp of the previous video frame of the candidate video frame to be analyzed and the estimated analysis time of the candidate video frame to be analyzed.

[0039] Among them, the previous video frame can be understood as a candidate video frame that is adjacent to the candidate video frame to be analyzed in the candidate video frame queue and has been analyzed and processed. The estimated analysis time can be the average time for analyzing a candidate video frame. The specific acquisition method includes but is not limited to obtaining it according to the time of analyzing and processing the candidate video frame that has been analyzed and processed, or estimating it according to the analysis and processing capabilities of the video playback device, or pre-setting it. The analysis virtual timestamp can be represented by Tn, Tn=Tnv+t2, where Tnv represents the analysis virtual timestamp of the previous video frame, and t2 represents the estimated analysis time of the candidate video frame to be analyzed.

[0040] Step S22: In response to the analysis that the relationship between the virtual timestamp and the system timestamp of the video playback device meets a preset analysis condition, determining to perform analysis processing on the candidate video frame to be analyzed.

[0041] In some application scenarios, in response to the analysis virtual timestamp being less than the system timestamp of the video playback device, it is determined that the relationship between the analysis virtual timestamp and the system timestamp of the video playback device meets the preset analysis condition. Or in some application scenarios, more stringent preset analysis conditions can also be set, such as the interval between the analysis virtual timestamp and the system timestamp must be greater than the preset timestamp interval. Among them, decoding the bitstream and analyzing and processing the candidate video frames are two independent processing processes, so the system timestamps when the two processes are executed may be inconsistent, which is normal.

[0042] In some application scenarios, in response to the analysis that the relationship between the virtual timestamp and the system timestamp of the video playback device does not meet the preset analysis conditions, the candidate video frame to be analyzed is not analyzed and processed, and the judgment can be continued or judged after a period of time whether the relationship between the virtual timestamp and the system timestamp of the video playback device meets the preset analysis conditions.

[0043] In some embodiments, the relevant information about the target object includes a local image corresponding to the target object. The relevant information about the target object may specifically be the YUV information of the local image. The above-mentioned method for determining the target video frame based on the relevant information about the target object in each candidate video frame and the relevant information about the target object in the frozen video frame may be: respectively obtaining the score of the target object in each local image. The score of the target object is related to one or more of the clarity, angle and posture of the target object. Then, from the local images of each candidate video frame and the local images containing the target object in the frozen video frame, a local image whose score meets the score condition is selected as the target local image, and the video frame corresponding to the target local image is the target video frame.

[0044] Exemplarily, for each partial image, a clarity score can be determined according to the clarity of the target object in the partial image, an angle score can be determined according to the angle of the target object, and a posture score can be determined according to the posture of the target object. The clarity score, angle score, and posture score are weighted to obtain the total score of each partial image. According to the total score of each partial image, the target partial image is determined. Exemplarily, the partial image with the highest total score can be used as the target partial image. The partial images described here include partial images of the target object in each candidate video frame and partial images of the target object in the frozen video frame.

[0045] In some embodiments, the image search method may further include the following steps: displaying a plurality of detection frames in the frozen video frame, wherein each detection frame corresponds to a candidate object; in response to receiving a selection instruction, taking the detection frame corresponding to the selection instruction as the target detection frame and the candidate object in the target detection frame as the target object, and the image area covered by the target detection frame in the frozen video frame as the local image containing the target object in the frozen video frame.

[0046] That is, after freezing the currently displayed video frame, several detection frames obtained by performing target detection on the frozen video frame continue to be displayed on the video frame, the selected detection frame is used as the target detection frame, the object within the target detection frame is used as the target object, and the area within the detection frame is used as the local image containing the target object in the above-mentioned frozen video frame.

[0047] In some embodiments, see Figure 4 , the above step S14 may include the following steps: Step S141: extracting features of a local image containing a target object in a target video frame to obtain target features.

[0048] Methods of feature extraction include, but are not limited to: using a feature extraction model to extract features or directly extracting data such as brightness changes and grayscale changes in a local image as target features.

[0049] Step S142: Obtaining storage features of several original images from the database.

[0050] The storage-in features may be storage-in features obtained when each original image is added to the database. The specific method of obtaining the storage-in features from the original image may be the same as the method of obtaining the target features from the local image.

[0051] Step S143: Compare the target feature with each stored feature to determine the original image that matches the target video frame.

[0052] The comparison method can be to perform feature matching to obtain the similarity between the target feature and each stored feature, and the original image corresponding to the maximum similarity can be used as the original image matching the target video frame, or a series of original images with large similarity can be used as the original image matching the target video frame.

[0053] Step S144: displaying the matched original image on the video playback device.

[0054] The method of displaying the matched original image on the video playback device includes but is not limited to: replacing the currently displayed video frame with the matched original image, or displaying the matched original image in a floating window on the video playback device.

[0055] In some embodiments, the image search method may further include the following steps: in response to receiving an image search instruction, continuing to play the video on the video playback device.

[0056] If the video is a real-time browsing video, the video data within the time period from receiving the freeze command to receiving the image search command has been lost. However, if the video is a playback video, the video data within the time period from receiving the freeze command to receiving the image search command can continue to be obtained and can continue to be played.

[0057] In some embodiments, in order to better understand the image search method proposed in the present application, the video playback process of the video playback device is as follows: Step 1: After the code stream is output, the real-time preview mode or the playback mode can be determined according to the actual business. If it is the real-time preview mode, execute step 3, otherwise execute step 2.

[0058] Step 2: Read as much video bitstream as possible into the cycle buffer according to the cycle buffer margin of each decoding channel to ensure that there is bitstream to be decoded in the cycle buffer.

[0059] Step three, the code stream parsing module parses the input code stream frame by frame to parse out key information such as frame coding type, image width, height, etc. The image width, height and other information can be used to subsequently determine the estimated decoding time. Among them, the code stream parsing module can be understood as a logical functional unit that performs frame-by-frame parsing of the input code stream in the image search method. The same applies to the following modules. The division of each module in this embodiment is relatively independent of the division of each unit in the image search device.

[0060] Step 4: According to the cache management strategy, the parsed raw code stream frames are sequentially cached to the video queue (VideoQueue) for management. Among them, the cache management strategy can solve the problem of network fluctuations affecting playback.

[0061] Step 5: The frame sending control module calculates the virtual timestamp for sending to display according to the status of the multi-level cache queue, and delivers the frame to be decoded to the shared queue to achieve smooth video playback.

[0062] Step 6: Using the system timeline as a benchmark and combining it with the frame sending control strategy, fine-tune the timestamp for sending and decoding. If the frame sending time is met, the frame to be decoded is sent to the decoding module; if the frame sending time is not met, wait for a cycle before making another judgment.

[0063] Step 7: The decoding module decodes the raw code stream frames in the queue to be decoded and outputs YUV.

[0064] Step 8: YUV is sent to intelligent analysis for analysis and processing, or sent to display output.

[0065] In some application scenarios, after receiving the freeze command, the freeze search mode is entered, and the video playback device can switch from normal decoding to full-speed decoding, and the multi-level cache in the decoding link is used to quickly decode the YUV information of each video frame, and analyze and process it to obtain the YUV information of each video frame about the target object, and then select the better video frame as the original image for image search. Specifically, the time difference between the time when the freeze command is received and the time when the search command is received is used to quickly decode and analyze the multi-level cache code stream in the decoding link to obtain more video frames about the target object, and then select the better video frame for image search.

[0066] In some application scenarios, the full-speed decoding process can be: First, the current decoding capacity margin of the device is calculated in real time, and the margin is marked as M 1080P@30fps. The average time to decode one frame is t1 = 1000 / (M*30*1920*1080 / W*H*fps). W represents the width of the bitstream, H represents the height of the bitstream, fps represents the frame rate of the bitstream, and 1000 represents 1000ms.

[0067] Then, the code stream frame management determines the decoding virtual timestamp Tdts of the code stream to be decoded. Tdts = Tdv +1000 / (M*30*1920*1080 / W*H*fps), where Tdv represents the decoding virtual timestamp of the previous frame. That is, Tdts=Tdv+t1.

[0068] Then, if Ts>Tdts and the candidate video frame queue is not full, the frame is sent to the decoding module. If the conditions are not met, the frame falls back to the code stream queue to be decoded in the cache and continues to judge after sleeping.

[0069] Next, the decoding module is responsible for decoding, inputting the bitstream frame and outputting the decoded YUV information.

[0070] The average time t2 required to process one frame is analyzed and counted in real time.

[0071] Then, the YUV frame management module determines the analysis virtual timestamp Tn of the next candidate video frame to be analyzed. Tn = Tnv + t2, where Tnv represents the analysis virtual timestamp of the previous frame.

[0072] If Ts>Tn, the candidate video frame is sent to the intelligent module for analysis and processing. If the conditions are not met, the frame falls back to the candidate video frame queue, and then continues to judge after a period of sleep.

[0073] The intelligent analysis module analyzes and processes the candidate video frames to obtain a local image of the target object contained in each candidate video frame.

[0074] A target local image is selected from each local image and a video frame corresponding to the target local image is used as a target video frame, and then image search processing is performed.

[0075] See also Figure 5 , Figure 5 It is a structural diagram of an embodiment of the image search device of the present application. The image search device 30 can execute the above-mentioned image search method. The image search device 30 includes a freezing unit 31, a video decoding unit 32, a video analysis unit 33 and an image search unit 34; the freezing unit 31 is used for the video playback device to freeze the video frame currently displayed by the video playback device in response to receiving a freeze instruction for the video being played; the video decoding unit 32 is used to decode at least part of the code stream about the video in the cache of the video playback device to obtain a number of candidate video frames that are not displayed; the video analysis unit 33 is used to determine the target video frame from the frozen video frame and a number of candidate video frames in response to receiving the image search instruction; the image search unit 34 is used to use the target video frame to perform image search processing.

[0076] In the above scheme, if the video playback device receives a freeze command for the video during the video playback process, it will first freeze the current display screen, and then decode the code stream in the cache to obtain several candidate video frames. When a search image command is received, the target video frame for image search can be determined from multiple video frames, which simplifies the image search process and compared with directly using the frozen video frame for image search, this scheme determines the target video frame from multiple video frames for image search processing, which can improve the accuracy of image search.

[0077] Among them, the functions of each module can be found in the description of the image search method embodiment, and will not be repeated here.

[0078] See also Figure 6 , Figure 64 is a schematic diagram of a structure of an embodiment of a video playback device of the present application. The video playback device 40 includes a memory 41 and a processor 42, and the processor 42 is used to execute program instructions stored in the memory 41 to implement the steps in any of the above-mentioned image search method embodiments. In a specific implementation scenario, the video playback device 40 may include but is not limited to: a microcomputer, a server, and in addition, the video playback device 40 may also include a carrier device such as a laptop computer and a tablet computer, which is not limited here.

[0079] Specifically, the processor 42 is used to control itself and the memory 41 to implement the steps in any of the above-mentioned image search method embodiments. The processor 42 can also be called a CPU (Central Processing Unit). The processor 42 may be an integrated circuit chip with signal processing capabilities. The processor 42 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 42 can be implemented by an integrated circuit chip.

[0080] In the above scheme, if the video playback device receives a freeze command for the video during the video playback process, it will first freeze the current display screen, and then decode the code stream in the cache to obtain several candidate video frames. When a search image command is received, the target video frame for image search can be determined from multiple video frames, which simplifies the image search process and compared with directly using the frozen video frame for image search, this scheme determines the target video frame from multiple video frames for image search processing, which can improve the accuracy of image search.

[0081] See also Figure 7 , Figure 7 The computer-readable storage medium 50 stores program instructions 51, which, when executed by a processor, implement the steps in any of the above-mentioned image search method embodiments.

[0082] In the above scheme, if the video playback device receives a freeze command for the video during the video playback process, it will first freeze the current display screen, and then decode the code stream in the cache to obtain several candidate video frames. When a search image command is received, the target video frame for image search can be determined from multiple video frames, which simplifies the image search process and compared with directly using the frozen video frame for image search, this scheme determines the target video frame from multiple video frames for image search processing, which can improve the accuracy of image search.

[0083] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0084] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0085] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. For example, units or components can be combined or integrated into another system, or some features can be ignored or not executed. In another image position, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0086] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A method for searching images, characterized in that: include: The video playback device freezes the video frame currently displayed by the video playback device in response to receiving a freeze instruction for the video being played; Decoding at least a portion of the bitstream of the video in the buffer of the video playback device to obtain a number of candidate video frames that are not displayed; In response to receiving the image search instruction, determining a target video frame from the frozen video frame and the plurality of candidate video frames; The target video frame is used to perform image search processing.

2. The method according to claim 1, characterized in that: The decoding of at least a portion of the code stream of the video in the buffer of the video playback device to obtain a plurality of candidate video frames that are not displayed includes: The following steps are performed for each bitstream of the video in the cache: Determine a decoding virtual timestamp of the code stream to be decoded in the cache, where the decoding virtual timestamp is obtained by a decoding virtual timestamp of a previously decoded code stream of the code stream to be decoded and an estimated decoding time consumption of the code stream to be decoded; In response to the relationship between the decoding virtual timestamp and the system timestamp of the video playback device satisfying a preset decoding condition, it is determined to decode the code stream to be decoded to obtain a candidate video frame that is not displayed.

3. The method according to claim 2, characterized in that The method further comprises: Adding each of the candidate video frames to a candidate video frame queue in sequence; Analyze and process each of the candidate video frames in the candidate video frame queue to obtain relevant information about the target object in each of the candidate video frames; The determining of the target video frame from the frozen video frame and the plurality of candidate video frames comprises: The target video frame is determined based on the relevant information about the target object in each of the candidate video frames and the relevant information about the target object in the frozen video frame.

4. The method according to claim 3, characterized in that The analyzing and processing each of the candidate video frames in the candidate video frame queue to obtain relevant information about the target object in each of the candidate video frames includes: The following steps are performed on each of the candidate video frames in the candidate video frame queue respectively: Determine an analysis virtual timestamp of a candidate video frame to be analyzed in the candidate video frame queue, where the analysis virtual timestamp is obtained by the analysis virtual timestamp of a previous video frame of the candidate video frame to be analyzed and an estimated analysis time of the candidate video frame to be analyzed; In response to the relationship between the analysis virtual timestamp and the system timestamp of the video playback device satisfying a preset analysis condition, it is determined to perform analysis processing on the candidate video frame to be analyzed.

5. The method according to claim 3, characterized in that: The method further comprises: Remove the candidate image frames that have been analyzed and processed from the candidate video frame queue; In response to the relationship between the decoded virtual timestamp and the system timestamp of the video playback device satisfying a preset decoding condition, determining to decode the code stream to be decoded to obtain a candidate video frame that is not displayed includes: In response to the relationship between the decoded virtual timestamp and the system timestamp of the video playback device satisfying the preset decoding condition, and the number of candidate video frames contained in the candidate video frame queue does not reach the maximum capacity of the candidate video frame queue, it is determined to decode the code stream to be decoded to obtain candidate video frames that are not displayed.

6. The method according to any one of claims 2 to 5, characterized in that The method further comprises: Obtaining a decoding capability margin of the video playback device; In response to the decoding capacity margin satisfying the full-speed decoding condition, the decoding speed of the video playback device is controlled to switch from normal decoding to full-speed decoding, and the number of video frames obtained by full-speed decoding per unit time is greater than the number of video frames obtained by normal decoding.

7. The method according to any one of claims 3 to 5, characterized in that The relevant information about the target object includes a local image corresponding to the target object, and determining the target video frame based on the relevant information about the target object in each of the candidate video frames and the relevant information about the target object in the frozen video frame includes: Obtaining scores of target objects in each of the partial images respectively, where the scores of the target objects are related to one or more of the clarity, angle, and posture of the target objects; From the partial images of each of the candidate video frames and the partial images of the frozen video frame containing the target object, a partial image whose score meets a score condition is selected as a target partial image, and a video frame corresponding to the target partial image is the target video frame.

8. The method according to claim 7, characterized in that The method further comprises: Displaying a plurality of detection frames in the frozen video frame, each detection frame corresponding to a candidate object; In response to receiving a selection instruction, the detection frame corresponding to the selection instruction is used as a target detection frame and the candidate object within the target detection frame is used as the target object, and the image area covered by the target detection frame in the frozen video frame is used as a local image containing the target object in the frozen video frame.

9. The method according to any one of claims 3 to 5, characterized in that: The using the target video frame to perform image search processing includes: Extracting features of a local image containing the target object in the target video frame to obtain target features; Obtaining the stored features of several original images from the database; Compare the target feature with each of the stored features to determine an original image that matches the target video frame; The matched original image is displayed on the video playback device.

10. The method according to any one of claims 1 to 5, characterized in that The method further comprises: In response to receiving the image search instruction, continue playing the video on the video playback device.

11. A video playback device, characterized in that: The invention comprises a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the method according to any one of claims 1 to 10.

12. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images