Object description information determination method and device, electronic equipment and storage medium

Abstract

The invention provides an object description information determination method and device, electronic equipment and a storage medium. The method comprises the steps that M first video frames of a video are acquired, each first video frame comprises a target object, and M is a positive integer; using a first model to identify the target object in each of the first video frames to obtain a first set, the first set comprising attribute information of the target object contained in each of the first video frames, the attribute information being used for representing image features of the target object, the first model is a model used for identifying N image features of a target object; the first set is input into a second model for natural language processing, text description information of the target object is obtained, and the text description information is used for representing text description of the image features of the target object.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a method, device, electronic device and storage medium for determining object description information. Background Art

[0002] With the explosive growth of video content, the demand for understanding and analyzing video content is also growing. Among them, the description of objects in the video (such as people, animals, objects, etc.) is one of the important research directions, which can be applied to a variety of scenarios. For example, for the video content retrieval scenario: this scenario can retrieve the target video more accurately through the description information of the characters; for the video content understanding scenario: this scenario can more deeply understand the video content such as character relationships, event development, etc. through the description information of the characters; for the video and video content generation scenario: this scenario can generate more attractive videos and video content such as character introductions and plot summaries through the description information of the characters.

[0003] At present, the description information of an object is usually generated based on the basic information of the object in the video (such as color, position, action, etc.). However, the description information of the object determined in this way is not comprehensive enough, resulting in the possibility that the description information of multiple objects is the same. Therefore, there is a problem in the prior art that the accuracy of the generated object description information is poor. Summary of the invention

[0004] The embodiment of the present invention provides a method, device, electronic device and storage medium for determining object description information, so as to solve the problem that the accuracy of the generated object description information is poor in the prior art. The specific technical solution is as follows:

[0005] In a first aspect of the present invention, a method for determining object description information is first provided, the method comprising:

[0006] Obtain M first video frames of the video, each of the first video frames includes a target object, and M is a positive integer;

[0007] Using a first model to identify the target object in each of the first video frames, obtaining a first set, wherein the first set includes attribute information of the target object contained in each of the first video frames, wherein the attribute information is used to characterize image features of the target object, and the first model is a model for identifying N image features of the target object;

[0008] The first set is input into a second model for natural language processing to obtain text description information of the target object, where the text description information is used to represent a text description of the image features of the target object.

[0009] In a second aspect of the present invention, a device for determining object description information is provided, the device comprising:

[0010] An acquisition module, used to acquire M first video frames of a video, each of the first video frames includes a target object, and M is a positive integer;

[0011] a recognition module, configured to recognize the target object in each of the first video frames using a first model to obtain a first set, wherein the first set includes attribute information of the target object contained in each of the first video frames, wherein the attribute information is used to characterize image features of the target object, and the first model is a model for recognizing N image features of the target object;

[0012] A natural language processing module is used to input the first set into a second model for natural language processing to obtain text description information of the target object, wherein the text description information is used to represent a text description of the image features of the target object.

[0013] In the third aspect of the implementation of the present invention, there is also provided an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; the memory is used to store computer programs; the processor is used to implement any of the above-mentioned methods for determining the object description information when executing the program stored in the memory.

[0014] In a fourth aspect of the implementation of the present invention, a computer-readable storage medium is further provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer executes any of the above-mentioned methods for determining object description information.

[0015] In a fifth aspect of the implementation of the present invention, there is also provided a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the above-mentioned methods for determining object description information.

[0016] The embodiment of the present invention obtains M first video frames of a video, each of which includes a target object; uses a first model to identify the target object in each of the first video frames to obtain a first set, the first set includes attribute information of the target object contained in each of the first video frames, the attribute information is used to characterize the image characteristics of the target object, and the first model is a model for identifying N image characteristics of the target object; the first set is input into a second model for natural language processing to obtain text description information of the target object, the text description information is used to represent the text description of the image characteristics of the target object. In this way, by identifying the attribute information of the object in the video that is used to represent the image characteristics, and generating the description information of the object based on the attribute information, since the image characteristics are unique to the object and each object has different image characteristics, the accuracy of the generated description information of the object can be improved, and the accuracy of video processing based on the description information of the object in the video can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0018] Figure 1 A flowchart of a method for determining object description information in an embodiment of the present invention;

[0019] Figure 2 A schematic diagram of the structure of a device for determining object description information in an embodiment of the present invention;

[0020] Figure 3 The figure is a schematic diagram of the structure of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0022] See also Figure 1 , Figure 1 Flow chart of a method for determining object description information in an embodiment of the present invention. The method for determining object description information includes:

[0023] Step S101: acquiring M first video frames of a video, each of the first video frames including a target object, and M is a positive integer.

[0024] A video may include multiple video frames, and the playback duration of the video may be arbitrary, which is not specifically limited here. In some embodiments, such as a video generation scenario, the video generation model may be trained by annotating the description information of the object in the video. In this scenario, the playback duration of the video is usually relatively short. If the acquired video is a long video, the long video may be cut according to the object scenario to obtain multiple video clips with relatively short playback durations. For video clips with image content of the target object in the video frame, the target object may be described in text as the video of this embodiment.

[0025] You can obtain videos shot in real time, receive videos sent by other electronic devices, and obtain videos on video platforms.

[0026] The M first video frames may be video frames including the image content of the target object in the video, and the M first video frames may be all the video frames in the video. Alternatively, the video frames may be obtained by performing frame extraction on the video. During the frame extraction process, a target detection algorithm may be used to perform target detection on the video frames to extract the first video frames having the image content of the target object. Alternatively, the video frames may be further screened from the frame-extracted video frames according to the image area where the target object of the video frame is located, which is not specifically limited here.

[0027] The target object can refer to a person such as a face. In this scenario, the person's attribute information can be described, for example, the facial expression, age, whether the person wears glasses, etc. The target object can also be an object or an animal. In this scenario, the attribute information of the object or animal can be described, for example, the color, size, shape, etc. of the object can be described in text.

[0028] Step S102, using the first model to identify the target object in each of the first video frames to obtain a first set, the first set includes attribute information of the target object contained in each of the first video frames, the attribute information is used to characterize the image features of the target object, the first model is a model for identifying N image features of the target object, N is an integer greater than 1.

[0029] Image features can refer to unique characteristics that are different from other people or objects. For example, image features can be gender, age, occupation, body shape, expression, posture, appearance and other characteristics of the image personality. These are different from basic characteristics such as the location and action of the object.

[0030] In some embodiments, for a person, the image features may include the person's age, gender, hairstyle, expression, whether or not the person wears glasses, etc. For an object, the image features may include the object's color, size, shape, etc.

[0031] In some embodiments, the attribute information of the target object may include the person's age, gender, hairstyle, expression, whether wearing glasses, and other attribute information, which can be represented by a list. For example, the attribute information of the target object may be [age: 35, gender: female, hairstyle: short hair, expression: smiling, whether wearing glasses: yes].

[0032] In some embodiments, the first model may be one or more face attribute models, and the attribute information of the face in the first video frame may be determined by at least one face attribute model. The face attribute model may be a deep learning model, such as a face recognition algorithm based on deep learning such as FaceNet and ArcFace.

[0033] In some embodiments, multiple face attribute models can be used to respectively determine different attributes of the face in the first video frame, and different attributes represent different image features of the face. Each face attribute model can be a binary classification model or a multi-classification model, which can determine an attribute of the face in the first video frame. For example, five face attribute models can be used to respectively determine the image features of the face in the first video frame, such as age, gender, hairstyle, expression, and whether glasses are worn. That is, when face detection is performed on the first video frame to obtain the image area where the face in the first video frame is located, the first video frame and the position information representing the image area where the face in the first video frame is located can be input into the five face attribute models to respectively determine the five different face attributes of age, gender, hairstyle, expression, and whether glasses are worn.

[0034] In some embodiments, a face attribute model may be used to determine multiple attributes of a face in the first video frame, and the face attribute model may be a multi-classification model. When face detection is performed on the first video frame to obtain an image region where the face is located in the first video frame, the first video frame and position information indicating the image region where the face is located in the first video frame may be input into the face attribute model to perform recognition of N preset image features, where N may be 5, such as age, gender, hairstyle, expression, and whether glasses are worn.

[0035] In some embodiments, an existing or new face detection algorithm can be used to perform face detection on the first video frame. In some embodiments, a face detection algorithm based on deep learning can be used, such as a multi-task convolutional neural network (MTCNN), a target detection model (You Only LookOnce, YOLO) and other face detection algorithms.

[0036] When the attribute information of the target object in each first video frame is obtained, the attribute information of the target object can be stored in a database, so that the attribute information of the target object in each first video frame can be aggregated according to the timestamp information of the M first video frames to obtain a first set. In some embodiments, the attribute information of the target object in each first video frame can be aggregated in the order of the timestamps of the first video frames from small to large to obtain the first set.

[0037] In some embodiments, the first set may be a large list obtained by aggregating multiple lists of attribute information of the target object, which may be sorted according to the timestamp of the first video frame corresponding to the attribute information of the target object. For example, the first set may be {[age: 35, gender: female, hairstyle: short hair, expression: smile, whether wearing glasses: yes]; [age: 35, gender: female, hairstyle: short hair, expression: smile, whether wearing glasses: no]; [age: 35, gender: female, hairstyle: short hair, expression: cry, whether wearing glasses: no]}.

[0038] In the related art, simple feature extraction methods, such as those based on the extraction of basic features such as color and texture, can only identify basic information such as the location and action of the person, but cannot effectively identify the attribute information representing the image characteristics of the person, such as the person's age, gender, and expression, that is, the recognition error rate of these image features in the related art is high, resulting in the inability to deeply analyze the attribute information of the person, and the lack of depth in the description of the person. In this step, the facial attribute information representing the image characteristics of the person in each first video frame can be identified through the facial attribute model, so that the unique facial features representing the image characteristics of the person in the video can be accurately identified, so as to achieve the purpose of deeply analyzing the attribute information of the person.

[0039] Step S103: input the first set into a second model for natural language processing to obtain text description information of the target object, where the text description information is used to represent a text description of the image features of the target object.

[0040] The first set is machine-readable descriptive information such as a model, which lacks human readability and is difficult for users to understand. In this step, text description information of the target object is further generated based on the first set. The text description information is used to represent the text description of the image features of the target object, which is obtained based on natural language processing and is human-readable.

[0041] The second model can be a large language model, and the large language model can use a Transformer model such as BERT, GPT-3, etc. The second model needs to be trained before use. In some embodiments, a pre-trained large language model can be used and fine-tuned to enable it to have the ability to generate text descriptions based on facial attribute information. Accordingly, the first set can be input into the large language model for natural language processing, and the large language model can output text description information that is consistent with human understanding and reading.

[0042] In some embodiments, the text description information can be used to describe the static image features of the target object, where the static image features can refer to image features that do not change over time, such as the gender of the person. In some embodiments, the text description information can also be used to describe the dynamic image features of the target object, where the dynamic image features can refer to image features that change over time, such as the text description information can describe the changes in the expression of the person, the actions performed by the object, etc. For example, the text description information can describe the change of a person's face from crying to smiling, the change of an object from a static state to a moving state, and the change of an animal from lying down to standing up.

[0043] For example, if the video playback time is 3 seconds, and 3 first video frames are obtained from the video, namely, video frame 1, video frame 2, and video frame 3, the first set may be {[age: 35, gender: female, hairstyle: short hair, expression: smile, whether wearing glasses: yes]; [age: 35, gender: female, hairstyle: short hair, expression: smile, whether wearing glasses: no]; [age: 35, gender: female, hairstyle: short hair, expression: cry, whether wearing glasses: no]}. In this scenario, the generated text description information of the target object may be "The video describes a 35-year-old girl with short hair and glasses, whose expression is first smiling, then the girl takes off the glasses, and then the girl's expression changes from smiling to crying.". The text description information describes the dynamic image features, namely, the change process of the character's expression and the change process of whether the glasses are worn, and describes the static image features, namely, the hairstyle and age.

[0044] In this embodiment, M first video frames of a video are obtained, each of which includes a target object; the target object in each of the first video frames is identified using a first model to obtain a first set, the first set including attribute information of the target object contained in each of the first video frames, the attribute information is used to characterize the image characteristics of the target object, and the first model is a model for identifying N image characteristics of the target object; the first set is input into a second model for natural language processing to obtain text description information of the target object, the text description information is used to represent a text description of the image characteristics of the target object. In this way, by identifying the attribute information of the object in the video that is used to represent the image characteristics, and generating the description information of the object based on the attribute information, since the image characteristics are unique to the object and each object has different image characteristics, the accuracy of the generated description information of the object can be improved, and the accuracy of video processing based on the description information of the object in the video can be improved.

[0045] The position detection of the target object in this step is a prior art. The prior art usually uses a rectangular frame to detect the position and obtains the image area where the target object is located.

[0046] The number of video frames in a video is large, and the video frames are continuous. The attribute information of the target object in many video frames is similar. If the attribute information of the target object is identified for all video frames in the video stream, or the video is subjected to frame extraction, there will be many video frames with repeated image content of the target object, and the processing efficiency will be relatively low. In some embodiments, the video can be subjected to frame extraction to obtain a second video frame, and according to the image area where the target object is located in the second video frame, the first video frame is further screened from the second video frame to improve the efficiency of the attribute information processing of the target object. Optionally, the step S101 specifically includes:

[0047] Extracting a second video frame from the video at every preset time interval;

[0048] Identify the currently extracted second video frame to obtain a second set, where the second set includes the second video frame containing the target object;

[0049] Performing object detection on the target object of the second video frame in the second set to obtain an image region where the target object is located in the second video frame, where the image region is used to represent a position of the target object in the second video frame;

[0050] Based on the similarity of the image regions in two adjacent second video frames in the second set, the second video frames in the second set are screened to obtain the M first video frames;

[0051] When M is greater than 1, the similarity between the image regions where the target object is located in any two adjacent first video frames is less than or equal to a first preset value.

[0052] The preset time interval may be arbitrary. In some embodiments, the preset time interval may be 0.1 second, such as 10 second video frames may be equally spaced and extracted from the video per second.

[0053] For the extracted second video frame, an existing or new object recognition algorithm can be used to identify the currently extracted second video frame to obtain a second video frame containing the image content of the target object, thereby obtaining a second set. For example, the playback time of the video is 3 seconds, and frame extraction is performed on the video frame, and 10 second video frames can be extracted per second. Further, the second video frames containing the image content of the target object are identified from the 10 second video frames of the 1st second, the 10 second video frames of the 2nd second, and the 10 second video frames of the 3rd second, respectively, and the second video frames are aggregated in ascending order according to the timestamps of the second video frames to obtain a second set.

[0054] An object detection algorithm based on deep learning, such as a multi-task convolutional neural network (MTCNN), a target detection model (You Only Look Once, YOLO), or other object detection algorithms, can be used to perform object detection on each second video frame in the second set to obtain an image region where the target object is located in the second video frame.

[0055] Afterwards, the similarity of the image regions in two adjacent second video frames may be determined, so as to screen the second video frames in the second set based on the similarity of the image regions in two adjacent second video frames in the second set to obtain M first video frames.

[0056] In some embodiments, the image contents of the image area where the target object is located in two adjacent second video frames can be compared, and specifically, the pixel values ​​of the pixels in the image area where the target object is located in two adjacent second video frames can be compared to determine the similarity of the image area in the two adjacent second video frames. When the number of identical pixels in the image area where the target object is located in two adjacent second video frames is greater than a certain threshold, it is determined that the similarity of the image area where the target object is located in the two adjacent second video frames is greater than a first preset value, otherwise it is determined that the similarity of the image area where the target object is located in the two adjacent second video frames is less than or equal to the first preset value. Among them, the first preset value can be set according to actual conditions and is not specifically limited here.

[0057] In some embodiments, the similarity of the image area where the target object is located in two adjacent second video frames can be determined based on the overlap of the image area where the target object is located in two adjacent second video frames. In some embodiments, the intersection over union (IoU) of the image area where the target object is located in two adjacent second video frames can be determined. When the IoU is greater than the second preset value, it can be determined that the similarity of the image area where the target object is located in the two adjacent second video frames is greater than the first preset value. Otherwise, it is determined that the similarity of the image area where the target object is located in the two adjacent second video frames is less than or equal to the first preset value. Among them, the second preset value can be set according to the actual situation and is not specifically limited here.

[0058] It should be noted that, if the similarity of the image area where the target object is located in two adjacent second video frames is less than or equal to the first preset value, it means that the image contents of the target object in the two adjacent second video frames are similar, and if the similarity of the image area where the target object is located in two adjacent second video frames is greater than the first preset value, it means that the image contents of the target object in the two adjacent second video frames are not similar.

[0059] In the case where it is determined that the image contents of the target object in different adjacent second video frames in the second set are similar, a first video frame can be obtained by arbitrarily selecting from the different adjacent second video frames. For example, for the 10 second video frames of the first second, it is determined that the image contents of the target object in the first to fifth second video frames are similar, and it is determined that the image contents of the target object in the sixth to tenth second video frames are similar. Then, the first second video frame can be selected as the first video frame, and the sixth second video frame can be selected as the first video frame, that is, two first video frames are obtained by selecting from the 10 second video frames of the first second. For the 10 second video frames of the second second and the 10 second video frames of the third second, the screening method can be similar to the method of selecting the first video frame from the 10 second video frames of the first second, which will not be described in detail here.

[0060] Afterwards, each first video frame obtained by the screening is aggregated in ascending order of the timestamps of the first video frames to obtain M first video frames. In this way, the image contents of the target object in any two adjacent first video frames are not similar, so there is no need to repeatedly extract the attribute information of the target object from similar video frames, thereby improving the efficiency of processing the attribute information of the target object in the video.

[0061] Optionally, before screening the second video frames in the second set based on the similarity of the image regions in two adjacent second video frames in the second set to obtain the M first video frames, the method further includes:

[0062] When the overlap of the image area where the target object is located in two adjacent second video frames is greater than a second preset value, determining that the image area where the target object is located in the two adjacent second video frames is greater than the first preset value;

[0063] When the overlap of the image areas where the target object is located in two adjacent second video frames is less than or equal to the second preset value, it is determined that the image areas where the target object is located in the two adjacent second video frames are less than or equal to the first preset value.

[0064] In some embodiments, the overlap of the image area where the target object is located in two adjacent second video frames can be determined by determining the IoU of the image area where the target object is located in two adjacent second video frames. When the IoU of the image area where the target object is located in two adjacent second video frames is greater than the second preset value, it can be determined that the similarity of the image area where the target object is located in the two adjacent second video frames is greater than the first preset value. Otherwise, it is determined that the similarity of the image area where the target object is located in the two adjacent second video frames is less than or equal to the first preset value. Among them, the second preset value can be set according to actual conditions and is not specifically limited here.

[0065] In this way, the process of determining the similarity of the image regions where the target object is located in two adjacent second video frames can be simplified, thereby simplifying the process of acquiring the first video frame.

[0066] Optionally, the step S102 specifically includes:

[0067] Inputting the first video frame and the target position information into the first model respectively to perform an attribute extraction operation to obtain the attribute information of the target object contained in each first video frame, wherein the target position information is the position information of the image area where the target object is located in the first video frame;

[0068] Taking the attribute information of the target object included in the first video frame as a set element, determining the first set, wherein an arrangement order of each set element in the first set is determined based on timestamp information of the first video frame corresponding to the set element;

[0069] Among them, the attribute extraction operation includes: performing feature extraction on the first video frame according to the target position information corresponding to the first video frame to obtain a first feature vector, and the first feature vector is used to represent the image information of the target object contained in the first video frame; according to the first feature vector, N image features of the target object are identified using a multi-classification method to obtain attribute information of the target object.

[0070] The target position information may be obtained by performing target detection on the first video frame using the object detection algorithm provided above.

[0071] Among them, the first model can be a multi-classification model, which can reduce the use of models. By using only one model, the attribute information of the target object for N image features can be extracted in a multi-classification manner, thereby improving the recognition efficiency of the attribute information of the target object, and effectively reducing the error rate of the target object's attribute information recognition, providing a more accurate attribute description of the target object.

[0072] Afterwards, the attribute information of the target object in each first video frame may be aggregated in ascending order of the timestamps of the M first video frames to obtain a first set.

[0073] It should be noted that the first model needs to be trained before use. During the training process, a loss function such as a cross entropy loss function can be used to train the model so that it can recognize multiple image features of the target object at one time using a multi-classification method.

[0074] It should be noted that before the first video frame is input into the first model, the first video frame can be preprocessed, such as resizing and grayscale processing, etc. This can reduce the complexity of the first model processing, thereby improving the efficiency of attribute information processing of the target object.

[0075] Optionally, the natural language processing includes: performing text embedding on the attribute information of the target object contained in each of the first video frames to obtain M second feature vectors corresponding one-to-one to the M first video frames; performing self-attention mechanism processing on the second feature vectors to obtain semantic description information corresponding to each attribute information; performing semantic recognition based on the semantic description information corresponding to each attribute information and the timestamp information of the first video frame corresponding to each attribute information to obtain text description information of the target object.

[0076] The second model may include a text embedding module, a self-attention mechanism processing module and a semantic recognition module. For each first video frame, the text embedding module may be used to perform text embedding on the attribute information of the target object in the first video frame to obtain a second feature vector of the attribute information of the target object in the first video frame. The self-attention mechanism processing module may be used to perform self-attention mechanism processing on the second feature vector to learn the semantics of the attribute information of the target object to obtain semantic description information corresponding to each attribute information.

[0077] Afterwards, semantic recognition can be performed based on the semantic description information corresponding to each attribute information and the timestamp information of the first video frame corresponding to each attribute information, so as to contextually associate the semantic description information of the attribute information of the target object in the first video frame based on the timestamp information of the first video frame. In this way, the difference between the semantic description information of the first video frame and the semantic description information of the first video frame before its timestamp can be recorded, and correspondingly, the text description information of the target object can be obtained based on the difference, thereby realizing the text description of the static image features and dynamic image features of the target object.

[0078] In this embodiment, the attribute information of the target object can be converted into a human-readable text description through a large language model, and a more readable text description of the attribute information can be provided, thereby improving the feature readability of the target object in the video.

[0079] In some embodiments, attribute information of the target object in each first video frame is determined by a face attribute model, and text description information of the target object in the video is generated according to the attribute information of the target object by a large language model. In this way, through the combination of the face attribute model and the large language model, the error rate of attribute recognition of the target object can be effectively reduced, and a more accurate attribute description of the target object in the video can be provided.

[0080] Optionally, after step S103, the method further includes:

[0081] Respectively identifying the image content of the target object in each of the first video frames to obtain identity information of the target object in the video;

[0082] The target object is identified according to the identity information, and text description information of the target object with the identity identification is generated.

[0083] In some embodiments, for each first video frame, the identity information of the face image content in the first video frame can be identified using an existing or new face recognition algorithm according to the image area where the face is located in the first video frame. The identity information may include name, logo, etc.

[0084] In some embodiments, the identity information of the target object and the first set may be input into the second model for natural language processing, so as to identify the target object according to the identity information and generate text description information of the target object with the identity identification. For example, the obtained text description information may be "The video stream describes a 35-year-old girl named Zhang San with short hair and glasses. Her expression is first smiling, then the girl removes the glasses she is wearing, and then the girl's expression changes from smiling to crying."

[0085] In this way, richer and more specific text description information can be generated.

[0086] Optionally, after step S103, the method further includes:

[0087] Using the video and the text description information of the target object as training data for a third model, and training the third model;

[0088] The video is a label in the training data, the text description information of the target object is an input sample in the training data, and the third model is used to generate a video according to the input text description information of the target object.

[0089] Among them, the third model can be a video generation model, which can be a deep learning model.

[0090] The video may be used as a label in the training data of the third model, and the text description information of the target object in the video may be used as an input sample in the training data of the third model to train the third model.

[0091] In this embodiment, by identifying the attribute information of the target object in the video and generating text description information of the target object based on the attribute information of the target object, the third model is trained using the information as input samples, so that the third model has the ability to accurately reproduce videos with corresponding target object attributes based on the text description information of the target object, so as to generate better videos and meet user needs for videos.

[0092] It should be noted that in the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0093] like Figure 2 As shown, the embodiment of the present invention further provides a device 200 for determining object description information, including:

[0094] An acquisition module 201 is used to acquire M first video frames of a video, each of which includes a target object, and M is a positive integer;

[0095] The recognition module 202 is used to recognize the target object in each of the first video frames using a first model to obtain a first set, wherein the first set includes attribute information of the target object contained in each of the first video frames, wherein the attribute information is used to characterize the image features of the target object, and the first model is a model for recognizing N image features of the target object;

[0096] The natural language processing module 203 is used to input the first set into a second model for natural language processing to obtain text description information of the target object, where the text description information is used to represent a text description of the image features of the target object.

[0097] Optionally, the acquisition module 201 is specifically configured to:

[0098] Extracting a second video frame from the video at every preset time interval;

[0099] Identify the currently extracted second video frame to obtain a second set, where the second set includes the second video frame containing the target object;

[0100] Performing object detection on the target object of the second video frame in the second set to obtain an image region where the target object is located in the second video frame, where the image region is used to represent a position of the target object in the second video frame;

[0101] Based on the similarity of the image regions in two adjacent second video frames in the second set, the second video frames in the second set are screened to obtain the M first video frames;

[0102] When M is greater than 1, the similarity between the image regions where the target object is located in any two adjacent first video frames is less than or equal to a first preset value.

[0103] Optionally, the device further includes a determining module, wherein the determining module is configured to:

[0104] When the overlap of the image regions where the target object is located in two adjacent second video frames is greater than a second preset value, determining that the similarity of the image regions where the target object is located in the two adjacent second video frames is greater than the first preset value;

[0105] When the overlap of the image regions where the target object is located in two adjacent second video frames is less than or equal to a second preset value, it is determined that the similarity of the image regions where the target object is located in the two adjacent second video frames is less than or equal to the first preset value.

[0106] Optionally, the identification module 202 is specifically configured to:

[0107] Inputting the first video frame and the target position information into the first model respectively to perform an attribute extraction operation to obtain the attribute information of the target object contained in each first video frame, wherein the target position information is the position information of the image area where the target object is located in the first video frame;

[0108] Taking the attribute information of the target object included in the first video frame as a set element, determining the first set, wherein an arrangement order of each set element in the first set is determined based on timestamp information of the first video frame corresponding to the set element;

[0109] Among them, the attribute extraction operation includes: performing feature extraction on the first video frame according to the target position information corresponding to the first video frame to obtain a first feature vector, and the first feature vector is used to represent the image information of the target object contained in the first video frame; according to the first feature vector, N image features of the target object are identified using a multi-classification method to obtain attribute information of the target object.

[0110] Optionally, the natural language processing includes: performing text embedding on the attribute information of the target object contained in each of the first video frames to obtain M second feature vectors corresponding one-to-one to the M first video frames; performing self-attention mechanism processing on the second feature vectors to obtain semantic description information corresponding to each attribute information; performing semantic recognition based on the semantic description information corresponding to each attribute information and the timestamp information of the first video frame corresponding to each attribute information to obtain text description information of the target object.

[0111] Optionally, the device further comprises:

[0112] A training module, used for training the third model by using the video and the text description information of the target object as training data of the third model;

[0113] The video is a label in the training data, the text description information of the target object is an input sample in the training data, and the third model is used to generate a video according to the input text description information of the target object.

[0114] The embodiment of the present invention further provides an electronic device, such as Figure 3 As shown, it includes a processor 301 , a communication interface 302 , a memory 303 and a communication bus 304 , wherein the processor 301 , the communication interface 302 , and the memory 303 communicate with each other via the communication bus 304 .

[0115] Memory 303, used for storing computer programs;

[0116] The processor 301 is used to execute the program stored in the memory 303 to implement the following steps:

[0117] Obtain M first video frames of the video, each of the first video frames includes a target object, and M is a positive integer;

[0118] Using a first model to identify the target object in each of the first video frames, obtaining a first set, wherein the first set includes attribute information of the target object contained in each of the first video frames, wherein the attribute information is used to characterize image features of the target object, and the first model is a model for identifying N image features of the target object, where N is an integer greater than 1;

[0119] The first set is input into a second model for natural language processing to obtain text description information of the target object, where the text description information is used to represent a text description of the image features of the target object.

[0120] Optionally, when the computer program is executed by the processor 301, it is further used for:

[0121] Extracting a second video frame from the video at every preset time interval;

[0122] Identify the currently extracted second video frame to obtain a second set, where the second set includes the second video frame containing the target object;

[0123] Performing object detection on the target object of the second video frame in the second set to obtain an image region where the target object is located in the second video frame, where the image region is used to represent a position of the target object in the second video frame;

[0124] Based on the similarity of the image regions in two adjacent second video frames in the second set, the second video frames in the second set are screened to obtain the M first video frames;

[0125] When M is greater than 1, the similarity between the image regions where the target object is located in any two adjacent first video frames is less than or equal to a first preset value.

[0126] Optionally, when the computer program is executed by the processor 301, it is further used for:

[0127] When the overlap of the image regions where the target object is located in two adjacent second video frames is greater than a second preset value, determining that the similarity of the image regions where the target object is located in the two adjacent second video frames is greater than the first preset value;

[0128] When the overlap of the image regions where the target object is located in two adjacent second video frames is less than or equal to a second preset value, it is determined that the similarity of the image regions where the target object is located in the two adjacent second video frames is less than or equal to the first preset value.

[0129] Optionally, when the computer program is executed by the processor 301, it is further used for:

[0130] Inputting the first video frame and the target position information into the first model respectively to perform an attribute extraction operation to obtain the attribute information of the target object contained in each first video frame, wherein the target position information is the position information of the image area where the target object is located in the first video frame;

[0131] Taking the attribute information of the target object included in the first video frame as a set element, determining the first set, wherein an arrangement order of each set element in the first set is determined based on timestamp information of the first video frame corresponding to the set element;

[0132] Among them, the attribute extraction operation includes: performing feature extraction on the first video frame according to the target position information corresponding to the first video frame to obtain a first feature vector, and the first feature vector is used to represent the image information of the target object contained in the first video frame; according to the first feature vector, N image features of the target object are identified using a multi-classification method to obtain attribute information of the target object.

[0133] Optionally, the natural language processing includes: performing text embedding on the attribute information of the target object contained in each of the first video frames to obtain M second feature vectors corresponding one-to-one to the M first video frames; performing self-attention mechanism processing on the second feature vectors to obtain semantic description information corresponding to each attribute information; performing semantic recognition based on the semantic description information corresponding to each attribute information and the timestamp information of the first video frame corresponding to each attribute information to obtain text description information of the target object.

[0134] Optionally, when the computer program is executed by the processor 301, it is further used for:

[0135] Using the video and the text description information of the target object as training data for a third model, and training the third model;

[0136] The video is a label in the training data, the text description information of the target object is an input sample in the training data, and the third model is used to generate a video according to the input text description information of the target object.

[0137] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0138] The communication interface is used for communication between the above electronic device and other devices.

[0139] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Alternatively, the memory may also be at least one storage device located away from the aforementioned processor.

[0140] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0141] In another embodiment provided by the present invention, a computer-readable storage medium is provided, in which instructions are stored. When the computer-readable storage medium is executed on a computer, the computer executes the method for determining object description information described in any of the above embodiments.

[0142] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes the method for determining object description information described in any one of the above embodiments.

[0143] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0144] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0145] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0146] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A method for determining object description information, characterized in that: The method comprises: Obtain M first video frames of the video, each of the first video frames includes a target object, and M is a positive integer; Using a first model to identify the target object in each of the first video frames, obtaining a first set, wherein the first set includes attribute information of the target object contained in each of the first video frames, wherein the attribute information is used to characterize image features of the target object, and the first model is a model for identifying N image features of the target object, where N is an integer greater than 1; The first set is input into a second model for natural language processing to obtain text description information of the target object, where the text description information is used to represent a text description of the image features of the target object.

2. The method according to claim 1, characterized in that The obtaining of M first video frames of a video includes: Extracting a second video frame from the video at every preset time interval; Identify the currently extracted second video frame to obtain a second set, where the second set includes the second video frame containing the target object; Performing object detection on the target object of the second video frame in the second set to obtain an image region where the target object is located in the second video frame, where the image region is used to represent a position of the target object in the second video frame; Based on the similarity of the image regions in two adjacent second video frames in the second set, the second video frames in the second set are screened to obtain the M first video frames; When M is greater than 1, the similarity between the image regions where the target object is located in any two adjacent first video frames is less than or equal to a first preset value.

3. The method according to claim 2, characterized in that Before the second video frames in the second set are screened based on the similarity of the image regions in two adjacent second video frames in the second set to obtain the M first video frames, the method further includes: When the overlap of the image regions where the target object is located in two adjacent second video frames is greater than a second preset value, determining that the similarity of the image regions where the target object is located in the two adjacent second video frames is greater than the first preset value; When the overlap of the image regions where the target object is located in two adjacent second video frames is less than or equal to a second preset value, it is determined that the similarity of the image regions where the target object is located in the two adjacent second video frames is less than or equal to the first preset value.

4. The method according to any one of claims 1 to 3, characterized in that: The using the first model to identify the target object in each of the first video frames to obtain the first set includes: Inputting the first video frame and the target position information into the first model respectively to perform an attribute extraction operation to obtain the attribute information of the target object contained in each first video frame, wherein the target position information is the position information of the image area where the target object is located in the first video frame; Taking the attribute information of the target object included in the first video frame as a set element, determining the first set, wherein an arrangement order of each set element in the first set is determined based on timestamp information of the first video frame corresponding to the set element; Among them, the attribute extraction operation includes: performing feature extraction on the first video frame according to the target position information corresponding to the first video frame to obtain a first feature vector, and the first feature vector is used to represent the image information of the target object contained in the first video frame; according to the first feature vector, N image features of the target object are identified using a multi-classification method to obtain attribute information of the target object.

5. The method according to any one of claims 1 to 3, characterized in that: The natural language processing includes: performing text embedding on the attribute information of the target object contained in each of the first video frames to obtain M second feature vectors corresponding one-to-one to the M first video frames; performing self-attention mechanism processing on the second feature vectors to obtain semantic description information corresponding to each attribute information; performing semantic recognition based on the semantic description information corresponding to each attribute information and the timestamp information of the first video frame corresponding to each attribute information to obtain text description information of the target object.

6. The method according to any one of claims 1 to 3, characterized in that: After inputting the first set into the second model for natural language processing to obtain text description information of the target object, the method further includes: Using the video and the text description information of the target object as training data for a third model, and training the third model; The video is a label in the training data, the text description information of the target object is an input sample in the training data, and the third model is used to generate a video according to the input text description information of the target object.

7. A device for determining object description information, characterized in that: The device comprises: An acquisition module, used to acquire M first video frames of a video, each of the first video frames includes a target object, and M is a positive integer; a recognition module, configured to recognize the target object in each of the first video frames using a first model to obtain a first set, wherein the first set includes attribute information of the target object contained in each of the first video frames, wherein the attribute information is used to characterize image features of the target object, and the first model is a model for recognizing N image features of the target object; A natural language processing module is used to input the first set into a second model for natural language processing to obtain text description information of the target object, wherein the text description information is used to represent a text description of the image features of the target object.

8. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-6 when executing a program stored in a memory.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that The method comprises computer instructions, which implement the method according to any one of claims 1 to 6 when executed by a processor.

Images