Video searching method and device, electronic equipment and storage medium

By segmenting and extracting dynamic video data, the main form code book is constructed, and the problem of incomplete video data search is solved, effective matching and search of clip video data is achieved, and the efficiency of user information acquisition is improved.

CN119025710AActive Publication Date: 2024-11-26PING AN TECH (SHENZHEN) CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411149321.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-20
Publication Date
2024-11-26
Estimated Expiration
2044-08-20

AI Technical Summary

Technical Problem

The prior art is difficult to use incomplete video data for video searches, resulting in users being unable to effectively understand information such as insurance products.

Method used

By obtaining dynamic video data for video segmentation, image preprocessing and subject environment segmentation, a subject shape code book is constructed, dynamic features in fragment video data are extracted, and approximate features are found from the code book to realize the search for incomplete video data.

Benefits of technology

Effective search of incomplete video data is realized, ensuring that the video features match the target video features, and improving the efficiency of user information acquisition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119025710A_ABST
    Figure CN119025710A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a video searching method and device, electronic equipment and a storage medium, and belongs to the technical field of feature extraction and artificial intelligence. The method comprises the following steps: performing video segmentation on dynamic video data containing a target object to obtain an action image frame sequence; performing image preprocessing on the action image frame sequence to obtain a binary image frame sequence; performing image segmentation on the binarized image frame sequence to obtain a main body contour set; constructing a main body form codebook based on the main body contour set; in response to a video search request which is sent by a client and carries clip video data, extracting dynamic characteristics in the clip video data to obtain main body dynamic characteristics; searching features similar to the main body dynamic features from the main body form codebook to obtain a target main body form feature set; and performing video search on the clip video data based on the target subject morphological feature set. According to the embodiment of the invention, the purpose of performing video search by using the clip video data can be achieved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of feature extraction and artificial intelligence technology, and in particular to a video search method and device, an electronic device and a storage medium. Background Art

[0002] In the scenario of the financial field, users can promote their understanding of insurance products by watching introductory videos of insurance products. For example, users can increase their acquisition of information about auto insurance by watching introductory videos of auto insurance.

[0003] At present, video data can only have the effect of introducing insurance products when it is complete. However, due to various reasons such as the network, video data is often prone to partial missing and unplayable, resulting in users being unable to understand insurance products based on incomplete video data. Therefore, how to achieve the purpose of video search using fragmented video data has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a video search method and device, an electronic device and a storage medium, aiming to achieve the purpose of video search using fragment video data.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a video search method, which is applied to a server, and the method includes:

[0006] Acquire dynamic video data containing a target object, and perform video segmentation on the dynamic video data to obtain a motion image frame sequence;

[0007] Performing image preprocessing on the action image frame sequence to obtain a binary image frame sequence;

[0008] Performing subject environment segmentation on the binary image frame sequence to obtain a subject contour set of the target object;

[0009] Based on the subject contour set, construct a subject morphology codebook;

[0010] In response to a video search request carrying segment video data sent by a client, dynamic features in the segment video data are extracted to obtain subject dynamic features;

[0011] Searching for features similar to the subject dynamic features from the subject morphology code book to obtain a target subject morphology feature set;

[0012] Based on the target subject morphological feature set, a video search is performed on the segment video data.

[0013] In some embodiments, constructing a subject morphology codebook based on the subject outline set includes:

[0014] Performing three-dimensional Zernike moment calculation on the subject contour set to obtain a subject morphological feature set;

[0015] Performing feature clustering on the subject morphology feature set to obtain a subject morphology feature category;

[0016] The subject morphology codebook is generated based on the subject morphology feature category.

[0017] In some embodiments, the subject contour set includes a plurality of subject contours, and performing three-dimensional Zernike moment calculation on the subject contour set to obtain a subject morphological feature set includes:

[0018] Performing coordinate normalization processing on each of the subject contours to obtain normalized contour data;

[0019] Calculating the Zernike moment based on the normalized profile data;

[0020] The Zernike moments are transformed in three dimensions to obtain the main body morphological feature set.

[0021] In some embodiments, performing a video search on the segment video data based on the target subject morphological feature set includes:

[0022] Querying the dynamic video data of the target subject morphological feature set to obtain target video data;

[0023] Obtain the number of clicks on the target video data;

[0024] Based on the number of clicks, the target video data is sorted to obtain a target video sequence.

[0025] In some embodiments, performing subject environment segmentation on the binary image frame sequence to obtain a subject contour set of the target object includes:

[0026] Performing smoothing processing on the binary image frame sequence to obtain a smoothed image frame sequence, wherein the smoothed image frame sequence includes a plurality of smoothed image frames;

[0027] Using a preset operator to perform gradient calculation on each of the smoothed image frames to obtain a set of candidate subject contours;

[0028] Performing non-maximum suppression processing on the candidate subject contour set to obtain a suppressed subject contour set;

[0029] The suppressed subject contour set is edge-connected to obtain a subject contour set.

[0030] In some embodiments, the subject morphology codebook includes a plurality of subject morphology features, and searching the subject morphology codebook for a feature similar to the subject dynamic feature to obtain a target subject morphology feature set includes:

[0031] Calculating the feature similarity between the subject dynamic feature and each of the subject morphological features;

[0032] Based on the feature similarity and a preset similarity threshold, selecting features similar to the subject dynamic features from the subject morphological features to obtain a candidate morphological feature sequence;

[0033] Based on the feature similarity, the candidate morphological feature sequences are sorted to obtain the target subject morphological feature set.

[0034] In some embodiments, performing image preprocessing on the action image frame sequence to obtain a binary image frame sequence includes:

[0035] Performing grayscale transformation on the action image frame sequence to obtain a grayscale image frame sequence;

[0036] The grayscale image frame sequence is binarized to obtain the binarized image frame sequence.

[0037] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a video search device, the device comprising:

[0038] A video segmentation module is used to obtain dynamic video data containing a target object and perform video segmentation on the dynamic video data to obtain an action image frame sequence;

[0039] A video frame binarization module, used for performing image preprocessing on the action image frame sequence to obtain a binarized image frame sequence;

[0040] An edge detection module is used to perform subject environment segmentation on the binary image frame sequence to obtain a subject contour set of the target object;

[0041] A codebook construction module, used to construct a subject morphology codebook based on the subject contour set;

[0042] The feature extraction module is used to respond to the video search request sent by the client and carry the video data, extract the dynamic features in the video data, and obtain the main dynamic features.

[0043] A feature approximation search module is used to search for features similar to the subject dynamic features from the subject morphology code book to obtain a target subject morphology feature set;

[0044] The video search module is used to perform a video search on the segment video data based on the target subject morphological feature set.

[0045] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0046] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0047] The video search method and device, electronic device and storage medium proposed in the present application perform subject environment segmentation on the acquired dynamic video data containing the target object to obtain a subject contour set containing the target object, and then construct a subject morphology code book based on the subject contour set, so as to form a search database for video features. Furthermore, when a video search request carrying fragment video data is received from a client, the subject dynamic features in the fragment video data are extracted, and the approximate features of the subject dynamic features are searched from the subject morphology code book to obtain a target subject morphology feature set, thereby ensuring that the video features in the incomplete video match the video features of the target video, and then performing video search on the fragment video data based on the target subject morphology feature set, so as to achieve the purpose of performing video search using the fragment video data. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a flow chart of a video search method provided by an embodiment of the present application;

[0049] Figure 2 yes Figure 1 Flow chart of step S102 in FIG.

[0050] Figure 3 yes Figure 1 Flow chart of step S103 in FIG.

[0051] Figure 4 yes Figure 1 Flow chart of step S104 in FIG.

[0052] Figure 5 yes Figure 4 Flow chart of step S401 in FIG.

[0053] Figure 6 yes Figure 1 Flow chart of step S106 in FIG.

[0054] Figure 7 yes Figure 1 Flow chart of step S107 in FIG.

[0055] Figure 8 is a schematic diagram of the structure of a video search device provided in an embodiment of the present application;

[0056] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0060] First, some nouns involved in this application are analyzed:

[0061] Codebook: Codebooks are used to define similarity measures between different categories. For a new video to be identified, its feature vector can be compared with the vocabulary in the codebook to calculate the similarity. Common similarity measures include Euclidean distance, cosine similarity, etc. With the codebook, feature calculation and matching become more efficient, and similar actions can be found quickly. The codebook converts complex features in the video into a lower-dimensional representation, thereby simplifying the subsequent classification and search process. By continuously updating the codebook, new categories and features can be continuously incorporated to enhance the adaptability of the model.

[0062] In the scenario of the financial field, users can promote their understanding of insurance products by watching introductory videos of insurance products. For example, users can increase their acquisition of information about auto insurance by watching introductory videos of auto insurance.

[0063] At present, video data can only have the effect of introducing insurance products when it is complete. However, due to various reasons such as the network, video data is often prone to partial missing and unplayable, resulting in users being unable to understand insurance products based on incomplete video data. Therefore, how to achieve the purpose of video search using fragmented video data has become a technical problem that needs to be solved urgently.

[0064] Based on this, the embodiments of the present application provide a video search method and device, an electronic device and a storage medium, which are intended to achieve the purpose of video search using fragment video data.

[0065] The video search method and device, electronic device and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the video search method in the embodiments of the present application is described.

[0066] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0067] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0068] The video search method provided in the embodiment of the present application relates to the field of feature extraction and artificial intelligence technology. The video search method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the video search method, etc., but is not limited to the above forms.

[0069] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0070] Figure 1 is an optional flow chart of a video search method provided in an embodiment of the present application, the method is applied to a server, Figure 1 The method may include but is not limited to steps S101 to S107.

[0071] Step S101, acquiring dynamic video data containing a target object, and performing video segmentation on the dynamic video data to obtain a motion image frame sequence;

[0072] Step S102, performing image preprocessing on the action image frame sequence to obtain a binary image frame sequence;

[0073] Step S103, performing subject environment segmentation on the binary image frame sequence to obtain a subject contour set of the target object;

[0074] Step S104, constructing a subject morphology codebook based on the subject outline set;

[0075] Step S105, in response to the video search request carrying the segment video data sent by the client, extract the dynamic features in the segment video data to obtain the main dynamic features;

[0076] Step S106, searching for features similar to the subject's dynamic features from the subject's morphological code book to obtain a target subject's morphological feature set;

[0077] Step S107: performing a video search on the segment video data based on the target subject morphological feature set.

[0078] In steps S101 to S107 shown in the embodiment of the present application, the server performs video segmentation on the acquired dynamic video data containing the target object to obtain a sequence of action image frames of the target object, and then performs a series of image preprocessing on the action image frame sequence to obtain a binary image frame sequence containing only black and white pixels. Further, the subject and the environment in the above-mentioned binary image frame sequence are segmented to obtain a set of subject contours containing only the target object. Secondly, based on the subject contour set, a subject morphology code book of the target object is constructed. Finally, when a video search request carrying fragment video data is received from the client, feature extraction is performed on the fragment video data to obtain subject dynamic features, and then, based on the subject dynamic features, approximate features in the above-mentioned subject morphology code book are searched to obtain a set of target subject morphology features, and video search is performed on the fragment video data based on the set of target subject morphology features. Therefore, the present application performs subject environment segmentation on the acquired dynamic video data containing the target object to obtain a subject contour set containing the target object, and then constructs a subject morphology code book based on the subject contour set, thereby forming a search database for video features. Furthermore, when a video search request carrying fragment video data is received from a client, the subject dynamic features in the fragment video data are extracted, and the approximate features of the subject dynamic features are searched from the subject morphology code book to obtain a target subject morphology feature set, thereby ensuring that the video features in the incomplete video match the video features of the target video, and then performing video search on the fragment video data based on the target subject morphology feature set, thereby achieving the purpose of performing video search using the fragment video data.

[0079] In step S101 of some embodiments, the target object may be a moving object or a target text. For example, when the dynamic video data is an introduction video of a car insurance product, the target object may be text such as "car insurance" or "vehicle insurance". When the dynamic video data is a dance video, the target object may be a moving human body. Dynamic video data may be video data in which the target object in the video is constantly moving and producing action changes.

[0080] The present application can obtain the required dynamic video data of the target object through network search, and after obtaining the dynamic video data of the target object, extract frames of the dynamic video data to obtain action image frames of the dynamic video data, sort the action image frames to obtain an action image frame sequence.

[0081] In one application scenario, taking a product introduction video of auto insurance as an example, the server can search for auto insurance products on the Internet to obtain auto insurance product videos, screen the auto insurance product videos to obtain auto insurance dynamic video data, and then perform video segmentation on the auto insurance dynamic video data to obtain each frame of auto insurance image in the auto insurance dynamic video data, and then sort the auto insurance images to obtain a sequence of auto insurance image frames.

[0082] In another application scenario, taking dance videos as an example, the server searches for dances on the Internet to obtain dance videos, and then screens the dance videos to obtain a dance dynamic video in which the target object in the video continuously moves and produces changes in movements. Furthermore, by extracting the dance dynamic video frame by frame, a dance image frame sequence of the dance dynamic video can be obtained.

[0083] In step S102 of some embodiments, the action image frame sequence is mostly color images, and the computational complexity of processing color images is very huge, resulting in low processing efficiency of the action image frame sequence. Therefore, after obtaining the action image frame sequence, it is necessary to perform a series of image preprocessing on the action image frame sequence so that the action image frame sequence is converted into a binary image frame sequence containing only melanin and white pigment.

[0084] See also Figure 2 In some embodiments, step S102 may include but is not limited to steps S201 to S202:

[0085] Step S201, performing grayscale transformation on the action image frame sequence to obtain a grayscale image frame sequence;

[0086] Step S202, binarization is performed on the grayscale image frame sequence to obtain a binarized image frame sequence.

[0087] In step S201 of some embodiments, after obtaining the motion image frame sequence, it is first necessary to convert the color image in the motion image frame sequence into a grayscale image, that is, to perform a grayscale transformation on the motion image frame sequence, thereby reducing the difficulty of processing the motion image frame sequence.

[0088] In step S202 of some embodiments, after obtaining the grayscale image frame sequence, a pigment threshold is selected, and each pixel in the grayscale image frame sequence is traversed to obtain the grayscale value of each pixel. Then, pixel judgment is performed on each pixel based on the pigment threshold and the grayscale value to implement binarization processing of the grayscale image frame sequence and obtain a binary image frame sequence.

[0089] In detail, if the grayscale value of a pixel in the grayscale image frame sequence is greater than or equal to the pigment threshold, the pixel is set to 255 (white); if the grayscale value of a pixel in the grayscale image frame sequence is less than the pigment threshold, it is set to 0 (black).

[0090] In steps S201 to S202 shown in the present embodiment, the action image frame sequence is subjected to grayscale transformation to obtain a grayscale image frame sequence, and then the grayscale image frame sequence is subjected to binarization processing to obtain a binarized image frame sequence. This can reduce the computational difficulty of the action image frame sequence, thereby improving the processing efficiency of the action image frame sequence.

[0091] In step S103 of some embodiments, after obtaining a binary image frame sequence, the subject and the environment can be segmented according to the pixel values ​​of the pixels in the binary image frame sequence, thereby obtaining a subject contour set of the target object. It should be noted that the subject contour set includes multiple subject contours of the target object.

[0092] For details, see Figure 3 In some embodiments, step S103 may include but is not limited to steps S301 to S304:

[0093] Step S301, performing smoothing processing on the binary image frame sequence to obtain a smoothed image frame sequence, wherein the smoothed image frame sequence includes a plurality of smoothed image frames;

[0094] Step S302, using a preset operator to perform gradient calculation on each smoothed image frame to obtain a set of candidate subject contours;

[0095] Step S303, performing non-maximum suppression processing on the candidate subject contour set to obtain a suppressed subject contour set;

[0096] Step S304: edge-connect the suppressed subject contour set to obtain a subject contour set.

[0097] In step S301 of some embodiments, a Gaussian filter may be used to smooth the binary image frame sequence to obtain a smoothed image frame sequence including multiple smoothed image frames, thereby reducing the impact of noise on the binary image frame sequence and improving the accuracy of the binary image frame sequence.

[0098] In step S302 of some embodiments, the preset operator may be a pre-set Sobel operator. The present application uses the operator to calculate the gradient of each smoothed image frame, and then according to the calculated gradient, the gradient intensity and gradient direction of each pixel in the smoothed image frame can be obtained, and further, according to the gradient intensity and gradient direction of the pixel point, the contour of the target object can be roughly determined, and thus, a candidate subject contour set can be obtained.

[0099] In step S303 of some embodiments, after obtaining the candidate subject contour set, non-maximum suppression is performed on the pixel points on the candidate subject contours in the candidate subject contour set, thereby further narrowing the contour range of the target object and obtaining the suppressed subject contour set.

[0100] In detail, the present application determines whether to retain the gradient strength of a pixel point on the candidate subject contour by comparing the gradient strength of the pixel point with the adjacent pixel points. For example, if the gradient strength of the above-mentioned pixel point is greater than the gradient strengths of two adjacent pixel points, the gradient strength of the pixel point is retained; otherwise, the gradient strength of the pixel point is set to zero.

[0101] In step S304 of some embodiments, after obtaining the suppressed subject contour set, the pixel points are classified by setting a low gradient intensity threshold and a high gradient intensity threshold, and then the classified pixel points are judged for gradient retention through edge connection, thereby obtaining the subject contour set of the target object.

[0102] In detail, a low gradient intensity threshold T1 and a high gradient intensity threshold T2 are set, and pixels with gradient intensity greater than T2 are regarded as strong pixels, pixels with gradient intensity between T1 and T2 are regarded as weak pixels, and the gradient intensity of pixels with gradient intensity less than T1 is suppressed to zero. Furthermore, when a weak pixel is connected to a strong pixel, the gradient intensity of the weak pixel will be regarded as a valid gradient intensity and retained. If the weak pixel is not connected to the strong pixel, the gradient intensity of the weak pixel will be suppressed to zero. Finally, edge connection is performed based on the pixels with retained gradient intensity to obtain the main contour set of the target object.

[0103] In steps S301 to S304 shown in this embodiment, by performing smoothing, gradient calculation, non-maximum suppression processing and edge connection on the binary image frame sequence, a set of subject contours of the target object is obtained, which can improve the accuracy of the subject contours and avoid feature extraction errors caused by incorrect subject contour division.

[0104] In step S104 of some embodiments, the subject morphology codebook may be a collection of representative subject morphology features of the target object, wherein the subject morphology features may be used to encode the video data input by the client, thereby achieving efficient representation and classification of subject morphology features of different categories. It should be noted that the subject morphology codebook includes multiple subject morphology features.

[0105] The present application classifies the features of the target object according to the subject contour set of the target object, and then selects representative subject morphological features from each category and fills them into a pre-constructed code book, thereby obtaining a subject morphological code book.

[0106] For details, see Figure 4 In some embodiments, step S202 may include but is not limited to steps S401 to S403:

[0107] Step S401, performing three-dimensional Zernike moment calculation on the subject contour set to obtain a subject morphological feature set;

[0108] Step S402, performing feature clustering on the subject morphology feature set to obtain a subject morphology feature category;

[0109] Step S403: Generate a subject morphology codebook based on the subject morphology feature category.

[0110] In step S401 of some embodiments, the subject morphological features of each subject contour in the subject contour set are obtained by calculating the three-dimensional Zernike moment representation of each subject contour, and then the subject morphological features of each subject contour are sorted and integrated to obtain the subject morphological feature set.

[0111] See also Figure 5 In some embodiments, step S401 may include but is not limited to steps S501 to S503:

[0112] Step S501, performing coordinate normalization processing on each subject contour to obtain normalized contour data;

[0113] Step S502, calculating the Zernike moment based on the normalized profile data;

[0114] Step S503: Perform three-dimensional transformation on the Zernike moments to obtain a set of main body morphological features.

[0115] In step S501 of some embodiments, the coordinates of each subject contour are normalized using a normalization formula to obtain normalized contour data, so as to map each subject contour into a unit circle and make the subject contour meet the calculation requirements of the Zernike moment.

[0116] In step S502 of some embodiments, after the normalized contour data is obtained, the normalized contour data is substituted into a calculation formula of the Zernike moment to obtain the Zernike moment of the normalized contour data.

[0117] It should be noted that the data required for calculating the Zernike moment include the grayscale value of the normalized contour image (which can be represented by a binary image).

[0118] In step S503 of some embodiments, the three-dimensional Zernike moment can represent the main morphological features of the target object within the main contour. Therefore, by converting the above Zernike moment into the three-dimensional Zernike moment, a main morphological feature set of the target object can be obtained.

[0119] In steps S501 to S503 shown in the present embodiment, coordinate normalization processing is performed on the subject contours in the subject contour set to obtain normalized contour data, and then the Zernike moments of each normalized contour data are calculated. Finally, the Zernike moments are converted into three-dimensional Zernike moments, so that a subject morphological feature set of the target object can be obtained, and feature extraction of the target object within the subject contour is achieved, so that the extracted subject morphological features can effectively describe the shape and characteristics of the target object within the subject contour.

[0120] In step S402 of some embodiments, the subject morphology feature set is classified by performing feature clustering on the subject morphology features in the subject morphology feature set to obtain a subject morphology feature category.

[0121] It should be noted that the present application can perform feature clustering on the subject morphological features in the subject morphological feature set by using clustering methods such as K-means clustering, hierarchical clustering, and DBSCAN.

[0122] In step S403 of some embodiments, based on the subject morphology feature categories obtained by clustering, representative subject morphology features are selected from each subject morphology feature category and written into a pre-constructed code book, thereby generating a subject morphology code book.

[0123] In steps S401 to S403 shown in the present embodiment, the three-dimensional Zernike moments of the subject contour set are calculated to obtain a subject morphological feature set, and then based on the subject morphological feature set, the subject morphological features are clustered to determine the subject morphological feature category. Finally, based on the subject morphological feature category, representative subject morphological features are screened out and stored in a pre-constructed code book to obtain a subject morphological code book, thereby realizing the construction of a feature directory for complete dynamic video data, thereby providing a search basis for video search of incomplete videos.

[0124] In step S105 of some embodiments, the client may be a terminal that inputs segment video data for video search. The segment video data may be incomplete video data, such as a segment of a dance video, a segment of an insurance product introduction video, and the like.

[0125] After receiving a video search request carrying segment video data sent by a client, the server in the present application can obtain the main dynamic features of the segment video data by extracting features from the segment video data.

[0126] In step S106 of some embodiments, after obtaining the subject dynamic features of the segment video data, subject morphological features similar to the subject dynamic features are screened out from the subject morphological code book, and subject morphological features with high similarity are used as target subject morphological features to form a target subject morphological feature set.

[0127] See also Figure 6 In some embodiments, step S106 may include but is not limited to steps S601 to S603:

[0128] Step S601, calculating the feature similarity between the subject's dynamic feature and each subject's morphological feature;

[0129] Step S602, based on feature similarity and a preset similarity threshold, features similar to the subject's dynamic features are selected from the subject's morphological features to obtain a candidate morphological feature sequence;

[0130] Step S603: sorting the candidate morphological feature sequences based on feature similarity to obtain a target subject morphological feature set.

[0131] In step S601 of some embodiments, the subject dynamic feature is compared with each subject morphological feature, and the similarity between the subject dynamic feature and each subject morphological feature is calculated using a similarity algorithm to obtain the feature similarity between the subject morphological feature and the subject dynamic feature.

[0132] In step S602 of some embodiments, the preset similarity threshold may be the minimum feature similarity allowed in the candidate morphological feature sequence set in advance. It should be noted that the preset similarity threshold in the present application may range from 75% to 100%. By comparing the size relationship between the feature similarity between the subject morphological feature and the subject dynamic feature and the similarity threshold, the subject morphological features with feature similarity greater than the similarity threshold can be screened out from the subject morphological feature set, and the subject morphological features that meet the conditions are stored in the pre-constructed candidate feature sequence to obtain the candidate morphological feature sequence.

[0133] In step S603 of some embodiments, after obtaining the candidate morphological feature sequence, the main morphological features in the candidate morphological feature sequence are sorted from large to small according to feature similarity, so as to obtain a target main morphological feature set.

[0134] In steps S601 to S603 shown in the present embodiment, the feature similarity between the subject dynamic feature and each subject morphological feature is calculated by a similarity calculation formula, and then a candidate morphological feature sequence whose feature similarity is greater than the similarity threshold is screened out from the subject morphological features according to a pre-set similarity threshold. Finally, the candidate morphological feature sequences are sorted according to the feature similarity to obtain a target subject morphological feature set, which can realize feature search for the subject dynamic feature in the subject morphological code book, thereby finding the subject morphological feature that is closest to the subject dynamic feature of the clip video data, thereby improving the accuracy of video search.

[0135] In step S107 of some embodiments, after obtaining the target subject morphological feature set, video data is selected from the dynamic video data as the target video of the segment video data according to the target subject morphological feature set, thereby realizing video search for the segment video data.

[0136] For details, see Figure 7 In some embodiments, step S107 may include but is not limited to steps S701 to S703:

[0137] Step S701, querying the dynamic video data of the target subject morphological feature set to obtain target video data;

[0138] Step S702, obtaining the number of clicks on the target video data;

[0139] Step S703: sort the target video data based on the number of clicks to obtain a target video sequence.

[0140] In step S701 of some embodiments, the target video data can be obtained by querying the video data corresponding to the target subject morphological feature set in the dynamic video data.

[0141] It should be noted that the target video data may be one or more.

[0142] In step S702 and step S703 of some embodiments, the number of clicks may be the number of times the target video data is clicked by the user to play. In order to improve the watchability of the video data searched for the segment video data, it is often necessary to sort the searched target video data. Specifically, the sorting method of the target video data may be based on dimensions such as the target video data's playback volume, number of clicks, and video duration.

[0143] The present application can crawl the number of clicks on the target video data through a web crawler, and then clarify the order of the target video data according to the number of clicks on the target video data, thereby achieving the purpose of video search using the fragment video data.

[0144] In steps S701 to S703 shown in the present embodiment, the target video data is obtained by querying the video data corresponding to the target subject morphological feature set in the dynamic video data, and then the target video data is sorted according to the number of clicks on the target video data to obtain a target video sequence, which can improve the watchability of the target video data and improve the user experience comfort.

[0145] The present application performs subject environment segmentation on the acquired dynamic video data containing the target object to obtain a subject contour set containing the target object, and then constructs a subject morphology code book based on the subject contour set, thereby forming a search database for video features. Furthermore, when a video search request carrying fragment video data is received from a client, the subject dynamic features in the fragment video data are extracted, and the approximate features of the subject dynamic features are searched from the subject morphology code book to obtain a target subject morphology feature set, thereby ensuring that the video features in the incomplete video match the video features of the target video. Then, based on the target subject morphology feature set, a video search is performed on the fragment video data, thereby achieving the purpose of performing video search using the fragment video data.

[0146] See also Figure 8 The embodiment of the present application also provides a video search device, which can implement the above-mentioned video search method, and the device includes:

[0147] The video segmentation module 801 is used to obtain dynamic video data containing a target object, and perform video segmentation on the dynamic video data to obtain a motion image frame sequence;

[0148] The video frame binarization module 802 is used to perform image preprocessing on the action image frame sequence to obtain a binarized image frame sequence;

[0149] The edge detection module 803 is used to perform subject environment segmentation on the binary image frame sequence to obtain a subject contour set of the target object;

[0150] A codebook construction module 804 is used to construct a subject morphology codebook based on the subject outline set;

[0151] The feature extraction module 805 is used to extract dynamic features from the video segment data in response to the video search request sent by the client, and obtain the main dynamic features.

[0152] A feature approximation search module 806 is used to search for approximate features of the subject dynamic features from the subject morphology code book to obtain a target subject morphology feature set;

[0153] The video search module 807 is used to perform a video search on the segment video data based on the target subject morphological feature set.

[0154] The specific implementation of the video search device is substantially the same as the specific implementation of the above-mentioned video search method, and will not be described in detail herein.

[0155] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above video search method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0156] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0157] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0158] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the video search method of the embodiment of this application;

[0159] Input / output interface 903, used to implement information input and output;

[0160] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WI FI, Bluetooth, etc.);

[0161] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0162] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0163] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the above-mentioned video search method is implemented.

[0164] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0165] The video search method, video search device, electronic device and storage medium provided in the embodiments of the present application perform subject environment segmentation on the acquired dynamic video data containing the target object to obtain a subject contour set containing the target object, and then construct a subject morphology code book based on the subject contour set, so as to form a search database for video features. Furthermore, when a video search request carrying fragment video data is received from a client, the subject dynamic features in the fragment video data are extracted, and the approximate features of the subject dynamic features are searched from the subject morphology code book to obtain a target subject morphology feature set, thereby ensuring that the video features in the incomplete video match the video features of the target video, and then performing video search on the fragment video data based on the target subject morphology feature set, thereby achieving the purpose of performing video search using the fragment video data.

[0166] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0167] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0168] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0169] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0170] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0171] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0172] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0173] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0174] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0175] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0176] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A video search method, characterized in that: Applied to the server, the method includes: Acquire dynamic video data containing a target object, and perform video segmentation on the dynamic video data to obtain a motion image frame sequence; Performing image preprocessing on the action image frame sequence to obtain a binary image frame sequence; Performing subject environment segmentation on the binary image frame sequence to obtain a subject contour set of the target object; Based on the subject contour set, construct a subject morphology codebook; In response to a video search request carrying segment video data sent by a client, dynamic features in the segment video data are extracted to obtain subject dynamic features; Searching for features similar to the subject dynamic features from the subject morphology code book to obtain a target subject morphology feature set; Based on the target subject morphological feature set, a video search is performed on the segment video data.

2. The method according to claim 1, characterized in that: The step of constructing a subject morphology codebook based on the subject outline set includes: Performing three-dimensional Zernike moment calculation on the subject contour set to obtain a subject morphological feature set; Performing feature clustering on the subject morphology feature set to obtain a subject morphology feature category; The subject morphology codebook is generated based on the subject morphology feature category.

3. The method according to claim 2, characterized in that The subject contour set includes a plurality of subject contours, and the three-dimensional Zernike moment calculation is performed on the subject contour set to obtain a subject morphological feature set, including: Performing coordinate normalization processing on each of the subject contours to obtain normalized contour data; Calculating the Zernike moment based on the normalized profile data; The Zernike moments are transformed in three dimensions to obtain the main body morphological feature set.

4. The method according to claim 1, characterized in that: The performing video search on the segment video data based on the target subject morphological feature set includes: Querying the dynamic video data of the target subject morphological feature set to obtain target video data; Obtain the number of clicks on the target video data; Based on the number of clicks, the target video data is sorted to obtain a target video sequence.

5. The method according to claim 1, characterized in that The performing subject environment segmentation on the binary image frame sequence to obtain a subject contour set of the target object includes: Performing smoothing processing on the binary image frame sequence to obtain a smoothed image frame sequence, wherein the smoothed image frame sequence includes a plurality of smoothed image frames; Using a preset operator to perform gradient calculation on each of the smoothed image frames to obtain a set of candidate subject contours; Performing non-maximum suppression processing on the candidate subject contour set to obtain a suppressed subject contour set; The suppressed subject contour set is edge-connected to obtain a subject contour set.

6. The method according to any one of claims 1 to 5, characterized in that: The subject morphology codebook includes a plurality of subject morphology features, and searching the subject morphology codebook for features similar to the subject dynamic features to obtain a target subject morphology feature set includes: Calculating the feature similarity between the subject dynamic feature and each of the subject morphological features; Based on the feature similarity and a preset similarity threshold, selecting features similar to the subject dynamic features from the subject morphological features to obtain a candidate morphological feature sequence; Based on the feature similarity, the candidate morphological feature sequences are sorted to obtain the target subject morphological feature set.

7. The method according to any one of claims 1 to 5, characterized in that: The performing image preprocessing on the action image frame sequence to obtain a binary image frame sequence includes: Performing grayscale transformation on the action image frame sequence to obtain a grayscale image frame sequence; The grayscale image frame sequence is binarized to obtain the binarized image frame sequence.

8. A video search device, characterized in that: The device comprises: A video segmentation module is used to obtain dynamic video data containing a target object and perform video segmentation on the dynamic video data to obtain an action image frame sequence; A video frame binarization module, used for performing image preprocessing on the action image frame sequence to obtain a binarized image frame sequence; An edge detection module is used to perform subject environment segmentation on the binary image frame sequence to obtain a subject contour set of the target object; A codebook construction module, used to construct a subject morphology codebook based on the subject contour set; The feature extraction module is used to respond to the video search request sent by the client and carry the video data, extract the dynamic features in the video data, and obtain the main dynamic features. A feature approximation search module is used to search for features similar to the subject dynamic features from the subject morphology code book to obtain a target subject morphology feature set; The video search module is used to perform a video search on the segment video data based on the target subject morphological feature set.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the video search method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the video search method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Video synopsis object retrieval method based on image semantic annotation

    CN106055653A

  • Multi-view clustering method and system based on self-paced learning and view empowerment

    CN115601578A

  • Video processing method and device, electronic equipment and storage medium

    CN117593211A

  • Focusing method and apparatus, electronic device, and computer readable storage medium

    WO2021057652A1