Video clip scene description generation method, model training method, device and equipment

Abstract

The invention provides a video clip scene description generation method, a model training method, devices and equipment, and relates to the field of image processing, in particular to the field of computer vision and deep learning. According to the specific implementation scheme, the method comprises the following steps: clustering face images in a video clip to be processed to obtain at least one clustering center; visual features of at least one clustering center and a plurality of sampling frames are respectively obtained, wherein the plurality of sampling frames are image frames which are extracted from the video clip and represent the video content. Then, mapping processing of text features is carried out on each visual feature, and a target text feature corresponding to each visual feature is obtained; and performing fusion processing on the prompt text feature of the prompt text input by the user and the target text feature corresponding to each visual feature to obtain scene description of the video clip. And compared with a mode of manually summarizing and concluding video plots, the time required for generating the scene description is greatly shortened, and the production efficiency is improved.

Description

Technical Field

[0001] The present disclosure relates to computer vision and deep learning in the field of image processing, and in particular to a method for generating a scene description of a video clip, a model training method, an apparatus and equipment. Background Art

[0002] With the development of mobile Internet technology, users' demand for utilizing fragmented time is growing. Therefore, short video platforms have emerged as a key tool to meet this demand. As an important form of short video platforms, short videos are loved by the majority of users because of their short length and compact plot.

[0003] In the prior art, when producing short videos corresponding to long videos (such as TV series and movies), the production staff needs to spend a lot of time watching and understanding the content of the long video and summarizing the plot to generate a scene description. However, this method of manually generating scene descriptions is time-consuming and has the problem of low production efficiency. Summary of the invention

[0004] The present disclosure provides a method for generating scene descriptions of video clips, a model training method, an apparatus and a device.

[0005] According to a first aspect of the present disclosure, a method for generating a scene description of a video clip is provided, comprising:

[0006] Clustering the face images in the video clip to be processed to obtain at least one cluster center;

[0007] Respectively acquiring visual features of the at least one cluster center and a plurality of sampling frames, where the plurality of sampling frames are image frames representing video content extracted from the video clip;

[0008] Mapping each visual feature to a text feature to obtain a target text feature corresponding to each visual feature;

[0009] The prompt text features of the prompt text input by the user and the target text features corresponding to each visual feature are fused to obtain a scene description of the video clip.

[0010] According to a second aspect of the present disclosure, a model training method is provided, comprising:

[0011] Sampling images in the video clip to obtain multiple sampling frames, and obtaining visual features corresponding to each sampling frame;

[0012] Extracting the conversation data in the video clip to obtain a plurality of text features corresponding to the video clip;

[0013] Obtaining a plurality of training samples according to the visual feature and the plurality of text features, each training sample comprising a visual feature and a calibrated text feature corresponding to the visual feature;

[0014] The multi-layer perceptron model is trained according to the multiple training samples to obtain a feature mapping model.

[0015] According to a third aspect of the present disclosure, a model training method is provided, comprising:

[0016] Obtain multiple scene descriptions for multiple video clips;

[0017] For each video clip, extracting features of a prompt text input by a user for the video clip to obtain at least one prompt text feature;

[0018] Extracting conversation data of the video clip, and acquiring at least one target text feature corresponding to the video clip based on the conversation data;

[0019] According to the scene description corresponding to each video clip, at least one prompt text feature and at least one target text feature, a plurality of training samples are obtained, each training sample including at least one prompt text feature corresponding to a video clip, at least one target text feature and a scene description;

[0020] The large language model is trained according to the multiple training samples to obtain a text fusion model.

[0021] According to a fourth aspect of the present disclosure, a model training method is provided, comprising:

[0022] Get multiple initial videos;

[0023] Slice each physical shot switching position in each initial video to obtain multiple target videos corresponding to each initial video;

[0024] According to the multiple initial videos and the multiple target videos corresponding to each initial video, a convolutional neural network model is trained to obtain a video segmentation model.

[0025] According to a fifth aspect of the present disclosure, a device for generating a scene description of a video clip is provided, comprising:

[0026] A first processing unit, configured to cluster the face images in the video clip to be processed to obtain at least one cluster center;

[0027] a second processing unit, configured to respectively obtain visual features of the at least one cluster center and a plurality of sampling frames, wherein the plurality of sampling frames are image frames extracted from the video clip and representing video content;

[0028] A third processing unit is used to perform text feature mapping processing on each visual feature to obtain a target text feature corresponding to each visual feature;

[0029] The fourth processing unit is used to fuse the prompt text features of the prompt text input by the user and the target text features corresponding to each visual feature to obtain the scene description of the video clip.

[0030] According to a sixth aspect of the present disclosure, a model training device is provided, comprising:

[0031] A sampling unit, used to sample images in the video clip to obtain multiple sampling frames, and obtain visual features corresponding to each sampling frame;

[0032] An extraction unit, used to extract the conversation data in the video clip to obtain a plurality of text features corresponding to the video clip;

[0033] A processing unit, configured to obtain a plurality of training samples according to the visual features corresponding to each sampling frame and the plurality of text features, wherein each training sample includes a visual feature and a calibrated text feature corresponding to the visual feature;

[0034] The training unit is used to train the multi-layer perceptron model according to the multiple training samples to obtain a feature mapping model.

[0035] According to a seventh aspect of the present disclosure, a model training device is provided, comprising:

[0036] An acquisition unit, used for acquiring multiple scene descriptions of multiple video clips;

[0037] A first extraction unit is used to extract features of the prompt text input by the user for each video clip to obtain at least one prompt text feature;

[0038] a second extraction unit, configured to extract the conversation data of the video segment, and obtain at least one target text feature corresponding to the video segment according to the conversation data;

[0039] A processing unit, configured to obtain a plurality of training samples according to a scene description corresponding to each video clip, at least one prompt text feature, and at least one target text feature, wherein each training sample includes at least one prompt text feature corresponding to a video clip, at least one target text feature, and a scene description;

[0040] The training unit is used to train the large language model according to the multiple training samples to obtain a text fusion model.

[0041] According to an eighth aspect of the present disclosure, a model training device is provided, comprising:

[0042] An acquisition unit, used for acquiring a plurality of initial videos;

[0043] A segmentation unit, used for segmenting each physical shot switching position in each initial video to obtain multiple target videos corresponding to each initial video;

[0044] A training unit is used to train a convolutional neural network model according to the multiple initial videos and the multiple target videos corresponding to each initial video to obtain a video segmentation model.

[0045] According to a ninth aspect of the present disclosure, there is provided an electronic device, including:

[0046] at least one processor; and

[0047] a memory communicatively connected to the at least one processor; wherein,

[0048] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any one of the methods described in the first aspect to the fourth aspect.

[0049] According to a tenth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any one of the methods described in the first to fourth aspects.

[0050] According to the eleventh aspect of the present disclosure, a computer program product is provided, the computer program product comprising: a computer program, the computer program being stored in a readable storage medium, at least one processor of an electronic device being able to read the computer program from the readable storage medium, the at least one processor executing the computer program so that the electronic device executes any one of the methods described in the first to fourth aspects.

[0051] The technology disclosed in the present invention solves the problem that manually generating scene descriptions is time-consuming and inefficient. By clustering the facial images in the video clip to be processed, at least one cluster center is obtained, and the visual features of at least one cluster center and multiple sampling frames are obtained respectively. The text feature mapping process is performed on each visual feature to obtain the target text feature corresponding to each visual feature; then, the prompt text feature of the prompt text input by the user and the target text feature corresponding to each visual feature can be fused to obtain the scene description of the video clip. In this way, the video scene description can be automatically generated, the time for producing the video scene description can be reduced, and the production efficiency can be improved.

[0052] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0054] Figure 1 It is a scene graph that can implement the video clip scene description generation method and model training method of the embodiment of the present disclosure;

[0055] Figure 2 is a schematic diagram of a first embodiment of the present disclosure;

[0056] Figure 3 It is a schematic diagram of visual features mapped to target text features;

[0057] Figure 4 is a schematic diagram of a second embodiment of the present disclosure;

[0058] Figure 5 is a schematic diagram of a third embodiment of the present disclosure;

[0059] Figure 6 is a schematic diagram of a fourth embodiment of the present disclosure;

[0060] Figure 7 is a schematic diagram of the disclosed clustering clusters;

[0061] Figure 8 is a schematic diagram of a fifth embodiment of the present disclosure;

[0062] Fig. 9 is a schematic diagram of a sixth embodiment of the present disclosure;

[0063] Fig.10 is a schematic diagram of a seventh embodiment of the present disclosure;

[0064] Fig.11 is a schematic diagram of an eighth embodiment of the present disclosure;

[0065] Fig.12 is a schematic diagram of a ninth embodiment of the present disclosure;

[0066] Fig.13 is a schematic diagram of a tenth embodiment of the present disclosure;

[0067] Fig.14 is a schematic diagram of an eleventh embodiment of the present disclosure;

[0068] Fig.15 is a schematic diagram of a twelfth embodiment of the present disclosure;

[0069] Fig.16 is a schematic diagram of a thirteenth embodiment of the present disclosure;

[0070] Fig.17 It is a block diagram of an electronic device used to implement the video clip scene description generation method and model training method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0071] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0072] With the rapid development of short video platforms, users can use fragmented time to complete content consumption of long videos by watching the description of the scene in the long video in the short video, and quickly understand the content information of the long video. This method not only meets the entertainment needs of users in their busy daily lives, but also enables users to obtain a large amount of valuable information in a short period of time, thereby improving the efficiency of information acquisition.

[0073] In the prior art, the process of converting a long video into a short video usually requires the production staff to spend a lot of time watching and understanding the content of the long video, and to summarize and generalize the plot in the long video in detail in order to generate the corresponding scene description. However, this method of relying on manual generation of scene description is not only time-consuming, but also significantly reduces production efficiency and has certain limitations.

[0074] In response to this existing problem, the technical conception process of the present disclosure is as follows: the inventor discovered in the process of describing the scenes of graduate students' growth videos that the production staff needs to spend a lot of time watching and understanding the content of long videos, as well as summarizing and generalizing the plots therein. This process is very time-consuming and affects the production efficiency. In addition, the process of manual summarization and generalization may be affected by personal understanding and preferences, resulting in a strong subjectivity of the scene description, which is not objective and comprehensive enough. In order to improve the efficiency of scene description in video production, the inventor considered whether it is possible to extract visual features in video clips and obtain the scene description of the video clips based on the prompt text entered by the user.

[0075] Based on the above technical conception process, the present disclosure provides a method for generating a scene description of a video clip, which can cluster the face images in the video clip to be processed, obtain the cluster center, and then obtain the visual features of multiple sampling frames and the cluster center respectively. Next, the text feature mapping process is performed on each visual feature to obtain the target text feature corresponding to each visual feature. Finally, the prompt text feature of the prompt text input by the user and the target text feature corresponding to each visual feature are fused to obtain the scene description of the video clip. By fusing the prompt text feature and the target text feature corresponding to the visual feature, the efficiency of the video scene description generation can be improved while reducing the production time, thereby achieving efficient and accurate video scene description generation.

[0076] Figure 1 is a scene graph that can implement the video clip scene description generation method and model training method of the embodiment of the present disclosure, such as Figure 1 As shown, the scenario at least includes a terminal device 101, an electronic device 102, and a server 103, wherein the terminal device 101, the electronic device 102, and the server 103 are communicatively connected with each other, the terminal device 101 is held by a user, and the user can operate the terminal device 101. The terminal device 101 can be a smart phone, a laptop computer, a tablet computer, or other device, and the electronic device 102 can also be a smart phone, a laptop computer, a tablet computer, or other device, and can also be a server.

[0077] Optionally, the communication connection between the above devices can be a wireless connection or a wired connection. The wired connection can be Ethernet, connecting the devices through a network cable (such as Cat5e, Cat6). It can also be serial communication, for example, including RS-232, RS-485 and other standards. It can also be fiber optic communication, such as a metropolitan area network (MAN) and a wide area network (WAN). Wireless communication can be a wireless local area network, Bluetooth, Zigbee, a cellular network, a low-power wide area network. For scenarios in remote areas, satellite communication can also be used.

[0078] The server 103 is used to obtain different training samples, train the multi-layer perceptron model, the large language model, and the convolutional neural network model respectively, obtain the feature mapping model, the text fusion model, and the video segmentation model, and send the trained model to the electronic device 102 for configuration.

[0079] The user sends the video to be processed to the electronic device 102 through the terminal device 101, and the electronic device 102 can obtain the video segment based on the video segmentation model; based on the feature mapping model, the visual features in the video segment are mapped to obtain the target text features corresponding to the visual features; the user sends the prompt text input by the user to the electronic device 102 through the terminal device 101, and the electronic device 102 can extract the features of the prompt text to obtain the prompt text features. Finally, the electronic device 102 can fuse the target text features with the prompt text features based on the text fusion model to generate a scenario description about the video segment.

[0080] The number of target text features obtained by the embodiment of the present disclosure can be determined according to a specific video clip, and the number of prompt text features obtained can be determined according to a specific prompt text.

[0081] Optionally, the server 103 may be a separate device or a part of the electronic device 102 .

[0082] The present invention provides a method for generating scene descriptions of video clips, a model training method, an apparatus and a device, which are applied to computer vision and deep learning in the field of image processing to improve the efficiency of scene description generation.

[0083] The following specific embodiments are used to describe in detail the technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0084] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information (such as facial images) involved are in compliance with the relevant laws and regulations and do not violate public order and good morals.

[0085] The following is an introduction to the specific implementation scheme of the method for generating scene description of video clips provided by the present disclosure.

[0086] Figure 2 is a schematic diagram of the first embodiment of the present disclosure, referring to Figure 2 The method for generating a scene description of a video clip shown in the figure can be configured as a device for generating a scene description of a video clip, which can be located in an electronic device, such as a mobile phone, a cloud server, a server cluster, a computer, a laptop computer, a tablet computer, etc. The specific type of the electronic device is not limited in the present disclosure. The method for generating a scene description of a video clip can include the following steps:

[0087] S201: Clustering facial images in a video clip to be processed to obtain at least one cluster center.

[0088] In this step, the face images in the video clip to be processed may be clustered, similar face images may be grouped into the same cluster, and then the face images in each cluster may be averaged to obtain at least one cluster center.

[0089] A video clip refers to a continuous video content cut out from a complete video. For example, a movie is three hours long, and a three-minute video clip can be cut out from it. This clip can include a battle scene or an emotional climax of an important character in the movie.

[0090] Optionally, the cluster center refers to the average feature of a group of face images, and the cluster center can be represented by the average face image. After the cluster center is determined, the cluster center can be named with a character code. For example, cluster center A can be named character code A.

[0091] S202: Obtain at least one cluster center and visual features of multiple sampling frames respectively.

[0092] In this step, sampling frames representing the video content can be extracted from the video clip as sampling frames, and visual features are extracted from at least one cluster center and multiple sampling frames obtained by clustering to obtain visual features of at least one cluster center and multiple sampling frames respectively.

[0093] Optionally, the sampling frame representing the video content may be determined by any one of the following methods:

[0094] Method 1: Extract sample frames from the video clip at fixed time intervals.

[0095] For example, the duration of a video clip is 4 seconds (s), and the video clip includes 40 image frames, namely, image frame 1, image frame 2, ..., and image frame 40. If the fixed time interval is 1 s, 4 sample frames can be extracted from the video clip. Optionally, the 4 sample frames can be image frame 1, image frame 11, image frame 21, and image frame 31.

[0096] Method 2: Extract sampling frames from video clips according to the changes in facial images.

[0097] For example, the duration of a video clip is 4 seconds (s), and the video clip includes 40 image frames, namely, image frame 1, image frame 2, ..., and image frame 40. If image frame 1 includes a face image A, image frame 5 includes a face image B, image frame 20 includes a face image C, and image frame 30 includes a face image D, then 4 sample frames can be extracted from the video clip. Specifically, the 4 sample frames can be image frame 1, image frame 5, image frame 20, and image frame 30.

[0098] Optionally, the change in the facial image may also include any one or a combination of a change in the number of faces, a change in the size of faces, and a change in the color of faces.

[0099] Method 3: Extract sample frames from video clips according to scene changes.

[0100] For example, the duration of a video clip is 4 seconds (s), and the 40 image frames included are image frame 1, image frame 2, ..., and image frame 40. If image frame 10, image frame 20, and image frame 30 correspond to different scenes, these sample frames with scene changes can be extracted from the video clip. Optionally, the 4 sample frames can be image frame 1 (initial scene), image frame 10 (scene change 1), image frame 20 (scene change 2), and image frame 30 (scene change 3).

[0101] Optionally, scene changes may also include any one or a combination of changes in objects, illumination, and background in the scene.

[0102] Optionally, the visual features may include but are not limited to facial feature points, texture information, color distribution, shape features, etc.

[0103] For example, after performing visual feature extraction on sample frame 1, visual feature 1 of sample frame 1 can be obtained, and visual feature 1 can be represented by visual feature vector 1 (i.e., the visual feature is saved as a visual feature vector representation). Specifically, if sample frame 1 is a cat, visual feature 1 representing the cat can be obtained, and visual feature 1 can be represented by visual feature vector 1.

[0104] Optionally, since the cluster center can be represented by the average features in a group of face images, when extracting visual features of the cluster center, feature extraction can be performed on the average features in the group of face images where the cluster center is located to obtain the visual features of the cluster center.

[0105] Similarly, the visual features of the cluster center can be represented by a visual feature vector. For example, when extracting visual features of cluster center A, the average features in a group of face images where cluster center A is located can be extracted to obtain visual features A of cluster center A, and visual features A can be represented by visual feature vector A.

[0106] S203: Perform text feature mapping processing on each visual feature to obtain a target text feature corresponding to each visual feature.

[0107] In this step, each visual feature can be mapped to a text feature through a feature mapping model, and each visual feature is input into the feature mapping model to obtain a corresponding target text feature output. The feature mapping model is a model for mapping visual features to text features obtained by training a multi-layer perceptron model.

[0108] Optionally, the text feature vector may be mapped to the visual feature vector corresponding to the visual feature of each image in the order of clustering centers first and then sampling frames to obtain the corresponding target text feature vector.

[0109] For example, the visual feature vector 1 corresponding to the visual feature 1 representing the cat can be input into the feature mapping model to obtain the target text feature vector 1 representing the cat.

[0110] Optionally, after the visual feature vector corresponding to the visual feature of the cluster center is input into the feature mapping model, the target text feature vector obtained can be an identifier about the cluster center, and the identifier can be represented by the target text feature vector. For example, after the visual feature vector A corresponding to the visual feature A of cluster center A is input into the feature mapping model, the target text feature vector A can be obtained, and the target text feature vector A is used to represent the identifier A of cluster center A. The identifier A can be a character code A.

[0111] S204: Fusing the prompt text features of the prompt text input by the user and the target text features corresponding to each visual feature to obtain a scene description of the video clip.

[0112] In this step, feature extraction can be performed on the prompt text input by the user to obtain prompt text features. The target text features corresponding to each visual feature and the obtained prompt text features are fused to obtain a scene description of the video clip.

[0113] The prompt text feature may include dividing the prompt text into smaller blocks with independent meanings, each block being a Token, and each Token may be a word, a partial word, or even a single letter.

[0114] Optionally, the prompt text input by the user may refer to a question about the content of the video clip. For example, prompt text 1 may be: "Please describe what happened in the video? The first picture is Zhang San". After feature extraction of the prompt text, 11 tokens may be obtained, namely: "Please, describe in detail, video, what happened?, first picture is Zhang San", then the corresponding prompt text feature 1 may be [Please, describe in detail, video, what happened?, first picture is Zhang San].

[0115] Likewise, each prompt text feature 1 may be represented by a prompt text feature vector 1. For example, the prompt text feature 1 may be represented by a prompt text feature vector 1.

[0116] For example, the prompt text feature [Please, describe, in detail, what, happened, in the video? The first picture, is, Zhang San] and the target text feature corresponding to each visual feature can be fused to obtain the scene description of the video clip as: Zhang San is playing with the cat. Among them, the first picture is the face image of cluster center A, and Zhang San is the identifier A of cluster center A.

[0117] The method for generating a scene description of a video clip provided in this embodiment clusters the face images in the video clip to be processed, determines at least one cluster center, and extracts the visual features of each cluster center and multiple sampling frames. Then, a text feature mapping is performed on each visual feature to generate a target text feature corresponding to each visual feature. The prompt text features of the prompt text input by the user and the target text features corresponding to each visual feature are fused to generate a scene description of the video clip. Compared with the method of manually summarizing and generalizing the video plot, the time required to generate the scene description is greatly reduced, and the production efficiency is improved. It also avoids the subjectivity of the production staff, which leads to the problem that the generated scene description is not objective and comprehensive enough.

[0118] Further, based on the above embodiment, in step S203, mapping processing is performed on each visual feature to a text feature to obtain a target text feature corresponding to each visual feature, including:

[0119] A feature mapping model is used to map text features to each visual feature to obtain target text features corresponding to each visual feature; wherein the feature mapping model is a model for mapping visual features and text features obtained by training a multi-layer perceptron model.

[0120] For example, visual feature 1 representing a cat can be input into a feature mapping model to obtain target text feature 1 representing a cat.

[0121] Optionally, in a specific implementation of mapping each visual feature to a text feature using a feature mapping model, the visual feature vector may be input into the feature mapping model to obtain a corresponding target text feature vector output.

[0122] For example, the visual feature vector 1 corresponding to the visual feature 1 of a cat is an M-dimensional vector. After the M-dimensional visual feature vector 1 is input into the feature mapping model, an N-dimensional target text feature vector 1 can be obtained.

[0123] It should be noted that mapping an M-dimensional vector to an N-dimensional vector means changing the dimension of the visual feature vector corresponding to the visual feature to achieve dimensional alignment with the target text feature vector corresponding to the target text feature, that is, the dimension of the visual feature vector can be reduced or increased. In addition, while aligning the dimensions, the eigenvalues ​​of different dimensions in the visual feature vector will also be processed by the feature mapping model to ensure that these eigenvalues ​​can be correctly reflected in the target text feature vector. Taking the dimensionality reduction of the visual feature vector as an example, the specific dimensions and eigenvalue changes can be shown in Table 1:

[0124] Table 1

[0125] Visual feature vector M Dimension Target text feature vector N-Dimension [2,25,30,…,8] 1024 [1,20,35,…,50] 768

[0126] It should be understood that if the M dimension is the same as the N dimension, when performing mapping processing after passing through the feature mapping model, it is only necessary to map the feature values ​​to achieve dimensional alignment.

[0127] Optionally, the sampling frame is usually a scene with relatively rich pictures, so the visual feature vector corresponding to the obtained visual feature can be a set composed of F M-dimensional vectors. When the visual feature vector is input into the feature mapping model to obtain the corresponding target text feature vector output, the F M-dimensional vectors can be mapped separately to obtain a set of F N-dimensional vectors. The set constitutes the target text feature vector, which is used to represent the target text feature.

[0128] For example, Figure 3 It is a schematic diagram of visual features mapped to target text features, such as Figure 3 As shown, after visual feature extraction is performed on the sampling frame, the obtained visual features include 2 sub-visual features (i.e., 2 M-dimensional vectors). The set of 2 M-dimensional vectors constitutes a visual feature vector. The 2 sub-visual features can be input into a feature mapping model to obtain sub-target text features corresponding to each sub-visual feature (i.e., 2 N-dimensional vectors). The set of 2 N-dimensional vectors constitutes a target text feature vector, which is used to represent the target text features.

[0129] The video clip scene description generation method provided in this embodiment, when mapping each visual feature to a text feature to obtain a target text feature corresponding to each visual feature, can convert each visual feature into a corresponding text feature through a feature mapping model, thereby obtaining the target text feature of each visual feature. By adopting the feature mapping model, it can be ensured that the details and characteristics of each visual feature are accurately captured and converted, thereby improving the accuracy of the generated text description. In addition, through accurate feature mapping, the generated text features can better reflect the semantic information of the original visual features, thereby making the generated scene description more natural and coherent, and improving the quality of text generation.

[0130] Further, based on the above embodiment, in step S204, the prompt text features of the prompt text input by the user and the target text features corresponding to each visual feature are fused to obtain a scene description of the video clip, including:

[0131] A text fusion model is used to fuse the prompt text features of the prompt text input by the user and the target text features corresponding to each visual feature to obtain a scene description of the video clip; wherein the text fusion model is a model for text feature fusion obtained by training a large language model.

[0132] Optionally, the prompt text input by the user can be extracted from the prompt text features by using a tokenizer component in natural language processing (NLP), the prompt text can be segmented into multiple tokens, and a sub-text feature vector corresponding to each token can be obtained. Each prompt text feature vector includes a sub-text feature vector corresponding to each token.

[0133] In a possible design, the dimensions of the text features and the target text features can be the same, which is a format that can be understood by the text fusion model.

[0134] Exemplarily, the prompt text input by the user can be subjected to feature extraction through a Tokenizer to obtain prompt text features, and the prompt text features and the target text features corresponding to each visual feature can be fused using a text fusion model to obtain a scene description of the video clip. In a specific implementation, a prompt text feature vector can be extracted through a Tokenizer, and the prompt text feature vector and the target text feature vector corresponding to each visual feature vector can be fused using a text fusion model to obtain a scene description of the video clip.

[0135] The method for generating a scene description of a video clip provided in this embodiment can use a text fusion model to fuse the prompt text features of the prompt text input by the user and the target text features corresponding to each visual feature to obtain the scene description of the video clip when fusing the prompt text features of the prompt text input by the user and the target text features corresponding to each visual feature to obtain the scene description of the video clip. By using the text fusion model, a large number of video clips can be processed in a relatively short time, and high-quality scene descriptions can be generated, thereby improving the efficiency of making scene descriptions. In addition, since the prompt text input by the user usually contains the user's specific concerns or needs for the video clip, by combining these prompt text features, a scene description that better meets the user's needs and interests can be generated, thereby improving user satisfaction.

[0136] In order to enable readers to more deeply understand the implementation principle of the present disclosure, the following Figure 4-Figure 9 right Figure 2 The illustrated embodiment is further refined.

[0137] Optionally, based on the above embodiment, before obtaining at least one cluster center, the video to be processed can also be physically segmented, and the same identifier can be set for at least two cluster centers whose facial feature similarity is higher than a preset value among the cluster centers corresponding to all the video clips obtained by segmentation.

[0138] Figure 4 is a schematic diagram of the second embodiment of the present disclosure, referring to Figure 4 The method for generating scene descriptions of video clips shown in the figure is different from the above-mentioned embodiment in that the present embodiment is to perform physical shot segmentation on the video to be processed, and further refine the method by setting the same identifier for at least two cluster centers whose facial feature similarity is higher than a preset value among the cluster centers corresponding to all the video clips obtained by segmentation. The corresponding refinement scheme may include the following steps:

[0139] S401, physically dividing a video to be processed into shots to obtain at least two video segments to be processed, where the dividing points between the video segments are locations for physical shot switching.

[0140] In this step, the video to be processed may be physically segmented according to the position of physical lens switching, and the video to be processed may be segmented into at least two video segments to be processed.

[0141] Optionally, the position of the physical lens switch refers to the transition point between different lenses in the video to be processed, and the position of the physical lens switch can be determined by scene changes, character changes, sudden changes in visual features, color histogram changes, etc.

[0142] For example, according to the fact that there are three scene changes in the video to be processed, the video to be processed can be physically segmented to obtain three video segments to be processed.

[0143] S402: among the cluster centers corresponding to all the video clips, at least two cluster centers having facial feature similarity higher than a preset value are set with the same identifier.

[0144] In this step, facial features in each video clip can be extracted, and then the facial feature similarity between all cluster centers is calculated. Next, a similarity threshold is set, and the cluster center pairs with similarities higher than the similarity threshold are set to the same identifier.

[0145] Optionally, face detection and feature extraction can be performed on the image frames in the video clip to obtain each face feature vector, and all face feature vectors are clustered to determine the number and position of cluster centers. The position of the cluster center represents the average feature of the cluster center in a group of face images.

[0146] Furthermore, according to a set similarity threshold, for cluster center pairs with a similarity higher than the threshold, it is considered that they represent similar facial features, and for cluster center pairs with a similarity higher than the threshold, the same identifier is set.

[0147] The method for generating a scene description of a video clip provided in this embodiment can physically segment the video to be processed to obtain at least two video clips to be processed, and set the same identifier for at least two cluster centers whose facial feature similarity is higher than a preset value among the cluster centers corresponding to all the video clips. By physically segmenting the video to be processed, the video to be processed can be divided into smaller and more coherent video clips, thereby effectively utilizing computing resources and reducing processing time.

[0148] Furthermore, the method for generating scene descriptions for video clips provided in this embodiment, by setting the same identifier for clustering centers of facial features with high similarity, helps to identify and monitor the appearance of the same person in different shot clips in the video, thereby enhancing the overall understanding of the video scene; in addition, by clustering and identifying similar facial features, subsequent processing steps such as scene description generation and character relationship analysis can be simplified, reducing repeated calculations and redundant information, and improving production efficiency.

[0149] On the basis of the above embodiment, in step S401, the video to be processed is physically segmented to obtain at least two video segments to be processed, including:

[0150] According to the video segmentation model, the video to be processed is physically segmented to obtain at least two video clips to be processed. The video segmentation model is a model that segments the video based on the physical lens switching position obtained by training a convolutional neural network model.

[0151] Optionally, if the lens switching position is a scene change, multiple initial videos can be obtained, and the scene changes in each initial video can be segmented to obtain multiple target videos corresponding to each initial video; then, the convolutional neural network model can be trained based on the multiple initial videos and the multiple target videos corresponding to each initial video to obtain a video segmentation model.

[0152] If the lens switching position is a sudden change in visual features, multiple initial videos can be obtained, and segmentation can be performed based on the sudden change in visual features (such as edges, textures) in each initial video to obtain multiple target videos corresponding to each initial video; then, the convolutional neural network model can be trained based on the multiple initial videos and the multiple target videos corresponding to each initial video to obtain a video segmentation model.

[0153] If the lens switching position is a color histogram change, multiple initial videos can be obtained, and segmentation is performed based on the color histogram change in each initial video to obtain multiple target videos corresponding to each initial video; then, the convolutional neural network model can be trained based on the multiple initial videos and the multiple target videos corresponding to each initial video to obtain a video segmentation model.

[0154] For example, the video segmentation model is a model that segments videos based on scene changes obtained through training of a convolutional neural network model. Then, according to the video segmentation model, the video to be processed is segmented based on scene changes to obtain at least two video clips to be processed.

[0155] It should be noted that if there is no physical shot switching position in the video to be processed, it means that the video to be processed is a complete shot and does not need to be segmented.

[0156] The video clip scene description generation method provided in this embodiment can perform physical shot segmentation on the video to be processed according to the video segmentation model to obtain at least two video clips to be processed. The video segmentation model is a model for segmenting the video based on the physical shot switching position obtained by training the convolutional neural network model. Through the video segmentation model, multiple shot switching features can be identified, thereby improving the accuracy and automation of segmenting the video to be processed, providing a good foundation for the subsequent generation of scene descriptions of video clips, making the generated scene descriptions more accurate, detailed and coherent.

[0157] Figure 5 is a schematic diagram of the third embodiment of the present disclosure, referring to Figure 5The method for generating a scene description of a video clip shown in the figure is different from the above-mentioned embodiment in that the present embodiment clusters the face images in the video clip to be processed to obtain further refinement of at least one cluster center, and the corresponding refinement scheme may include the following steps:

[0158] S501: extract multiple face images from the video clip using a face detection algorithm.

[0159] Exemplarily, the face detection algorithm can be a detection algorithm based on deep learning, such as a multi-task cascade convolutional neural network; it can also be an advanced detection algorithm such as an istogram of oriented gradients (HOG) or a target detection algorithm (Single Shot MultiBox Detector, SSD).

[0160] For example, based on a deep learning detection algorithm, six face images can be extracted from a video clip, namely face image A, face image B, face image C, face image D, face image E, and face image F.

[0161] S502: Cluster multiple face images based on the face features in each face image to obtain at least one cluster.

[0162] In this step, facial feature vectors of each facial image may be extracted, and cluster analysis may be performed on the facial feature vectors using a clustering algorithm, so that facial images with similar features may be classified into the same cluster.

[0163] Optionally, the facial feature vector of each face image can be obtained by using a feature extraction method such as a deep learning model, scale-invariant feature variation, local binary pattern, principal component analysis, etc. The deep learning model can be a convolutional neural network or a visual self-attention network (Vision Transformer, Vit).

[0164] Optionally, the facial feature vectors may be clustered using K-means, hierarchical clustering, or a density-based clustering method (DBSCAN).

[0165] For example, the facial feature vectors of 6 facial images can be extracted through the convolutional neural network, namely, facial feature vector A, facial feature vector B, facial feature vector C, facial feature vector D, facial feature vector E, and facial feature vector F; the 6 facial images can be classified into 2 clusters through the K-means clustering algorithm. Among them, cluster 1 can include facial image A, facial image C, and facial image E; cluster 2 can include facial image B, facial image D, and facial image F.

[0166] S503: Based on each cluster, obtain at least one cluster center.

[0167] In this step, after determining the facial images included in each cluster, the cluster center of the cluster can be determined by calculating the average value of all facial feature vectors in each cluster. The cluster center can be represented by the average facial image.

[0168] The method for generating a scene description of a video clip provided in this embodiment can extract multiple images containing faces from a video clip through a face detection algorithm. Then, based on the facial features in each face image, the multiple face images are clustered to obtain at least one cluster. Based on the face images in each cluster, the average value of the feature vector is calculated to determine the cluster center of the cluster. By extracting and clustering face images, the characters in the video clip can be more accurately identified, thereby generating a more accurate and detailed scene description; in addition, similar face images are classified into the same cluster, which simplifies the data management and analysis process and improves processing efficiency.

[0169] Figure 6 is a schematic diagram of the fourth embodiment of the present disclosure, referring to Figure 6 The method for generating a scene description of a video clip shown in the figure is different from the above-mentioned embodiment in that the present embodiment is based on each clustering cluster, and further refines at least one cluster center, and the corresponding refinement scheme may include the following steps:

[0170] S601: Obtain the initial cluster center of each cluster.

[0171] In this step, after determining the face images included in each cluster, the face feature vectors of the face images of the cluster can be extracted, and the initial cluster center of each cluster can be obtained by calculating the mean of the face feature vectors.

[0172] S602: For each cluster, respectively calculate the similarity between each face image in the cluster and the initial cluster center of the cluster.

[0173] In this step, the similarity between each face image in the cluster and the initial cluster center of the cluster can be calculated by a similarity measurement method to obtain the similarity between each face image and the initial cluster center of the cluster to which it belongs.

[0174] Optionally, similarity measurement methods may include Euclidean distance, cosine similarity, Manhattan distance, etc.

[0175] Exemplarily, the Euclidean distance is used as a similarity measurement method. For any cluster, the facial feature vector of each face image in the cluster can be extracted, and the Euclidean distance between the facial feature vector and the initial cluster center (the mean of the facial feature vector) is calculated.

[0176] S603: Select at least one target face image with the highest similarity to the initial cluster center from the clusters, and determine the cluster center of the cluster based on the at least one target face image.

[0177] In this step, for any cluster, after obtaining the similarity between each face image and the initial cluster center of the cluster to which it belongs, the initial cluster center can be updated to obtain an updated cluster center. In a specific implementation, at least one target face image with the highest similarity to the initial cluster center can be selected from the cluster, and the updated cluster center can be determined based on the at least one target face image.

[0178] Figure 7 is a schematic diagram of the disclosed clustering cluster, reference Figure 7 , Figure 7 The figure shows the similarity between each face image and the initial cluster center of the cluster in which it is located. The closer the distance between the face image and the initial cluster center of the cluster in which it is located, the higher the similarity.

[0179] For example, a cluster includes 6 face images, and face feature vector A, face feature vector B, face feature vector C, face feature vector D, face feature vector E, and face feature vector F can be extracted; if the initial cluster center of the cluster corresponds to the face feature vector C1, Euclidean distance can be used as the similarity measurement method to obtain distances L1, L2, L3, L4, L5, and L6. The smaller the distance, the higher the similarity.

[0180] Optionally, after determining the similarity between each face image and the initial cluster center of the cluster to which it belongs, the similarities can be sorted from high to low, and at least one target face image can be selected in order of similarity from high to low, and an updated cluster center can be determined based on the at least one target face image.

[0181] The video clip scene description generation method provided in this embodiment first obtains the initial cluster center of each cluster cluster. Then, for each cluster cluster, the similarity between each face image therein and the initial cluster center of the cluster cluster is calculated respectively. On this basis, at least one target face image with the highest similarity to the initial cluster center is selected from each cluster cluster, and the cluster center of the cluster cluster is determined based on these target face images. By selecting the target face image with the highest similarity to the initial cluster center to determine the final cluster center, the characteristics of the entire cluster cluster can be more accurately represented, thereby improving the accuracy of clustering. In addition, the final cluster center is determined based on the most similar target face image, and will not be affected by all face images, which can reduce the interference of noise data on the clustering results.

[0182] Figure 8 is a schematic diagram of the fifth embodiment of the present disclosure, referring to Figure 8 The method for generating a scene description of a video clip shown in the embodiment is different from the above-mentioned embodiment in that the embodiment is a further refinement of determining the cluster center of the cluster cluster according to at least one target face image, and the corresponding refinement scheme may include the following steps:

[0183] S801: If the number of target face images is one, use the target face image as the cluster center of the cluster.

[0184] Specifically, if the number of target face images is one, the face image with the highest similarity can be selected as the cluster center of the cluster according to the order of similarity from high to low.

[0185] S802: If the number of target face images is greater than one, taking the mean of at least two target face images as the cluster center of the cluster.

[0186] Specifically, if the number of target face images is greater than one, at least two face images with the highest similarity may be selected according to the order of similarity from high to low, and the mean of the at least two target face images may be used as the cluster center of the cluster.

[0187] The video clip scene description generation method provided by this embodiment can use the target face image as the cluster center of the cluster cluster when the number of target face images is one; and use the mean of at least two target face images as the cluster center of the cluster cluster when the number of target face images is greater than one. By flexibly selecting the number of target face images to determine the cluster center, the accuracy and adaptability of clustering can be improved, thereby enhancing the accuracy of the video clip scene description.

[0188] Next, based on any of the above embodiments, Fig. 9 , through specific examples, the above-mentioned method for generating scene description of video clips is further explained.

[0189] Fig. 9 is a schematic diagram of the sixth embodiment of the present disclosure, referring to Fig. 9 The method for generating a scene description of a video clip shown may include the following steps:

[0190] S901: Obtain a video to be processed.

[0191] For example, the electronic device may receive a video to be processed sent by a user through a terminal device.

[0192] S902: Physical shot segmentation and face clustering.

[0193] Optionally, the video to be processed may be physically segmented by a video segmentation model to obtain at least two video segments to be processed. For example, by physically segmenting the video to be processed by a video segmentation model, two video segments to be processed may be obtained, namely, video segment 1 and video segment 2.

[0194] Furthermore, the face images in the video clip to be processed may be clustered to obtain at least one cluster center. For example, a cluster center A may be obtained by clustering the video clip 1 to be processed.

[0195] S903: Acquire visual features.

[0196] Optionally, visual features of at least one cluster center and a plurality of sample frames may be extracted, wherein the plurality of sample frames are image frames representing video content extracted from video clips.

[0197] For example, Vit can be used to extract visual features of cluster center A and sampling frames in the video clip 1 to be processed to obtain 5 visual feature vectors, where cluster center A can be image A, and the sampling frames can include 4 image frames, namely image frame B, image frame C, image frame D, and image frame E. The 5 visual feature vectors are visual feature vector A corresponding to cluster center A, visual feature vector B corresponding to image frame B, visual feature vector C corresponding to image frame C, visual feature vector D corresponding to image frame D, and visual feature vector E corresponding to image frame E.

[0198] Specifically, Vit can obtain the visual feature vector of the sampled frame through the following steps:

[0199] Step ①: Divide the image with the input sampling frame size of H*W*C into X fixed-size P*P*C image blocks. Among them, H represents the number of pixels in the vertical direction of the image, in pixels (px); W represents the number of pixels in the horizontal direction of the image; C represents the number of color channels of the image; P represents the side length of each image block, that is, the height and width of each image block are P pixels.

[0200] For example, the size of the input sampling frame is 224*224*3, and the sampling frame can be divided into small blocks of 16*16*3, and a total of 196 image blocks can be obtained.

[0201] Step ②: Convert each image block into a one-dimensional vector of P*P*C dimensions, and then concatenate X one-dimensional vectors to form a two-dimensional vector of X*(P*P*C) to obtain the visual feature vector of the sampling frame.

[0202] For example, each image block can be converted into a 768-dimensional one-dimensional vector, and then 196 one-dimensional vectors can be concatenated to form a 196*768 two-dimensional vector to obtain the visual feature vector of the sampling frame.

[0203] S904: Mapping the visual features to text features to obtain target text features.

[0204] For example, the feature mapping model can be used to obtain the target text feature vector A corresponding to visual feature vector A, the target text feature vector B corresponding to visual feature vector B, the target text feature vector C corresponding to visual feature vector C, the target text feature vector D corresponding to visual feature vector D, and the target text feature vector E corresponding to visual feature vector E.

[0205] S905: Obtain prompt text features of the prompt text input by the user.

[0206] Optionally, the prompt text input by the user may be subjected to feature extraction by using a Tokenizer in natural language processing to obtain prompt text features.

[0207] For example, if the prompt text input by the user is: "Please describe what happened in the video? The first picture is Zhang San, and the rest are sample frames of the video. Please analyze them frame by frame", then the prompt text input by the user can be extracted through Tokenizer to obtain a prompt text feature vector used to represent the prompt text features.

[0208] S906: Fusing the prompt text feature and the target text feature corresponding to each visual feature to obtain a scenario description.

[0209] For example, based on the text fusion model, the prompt text feature vector used to represent the prompt text feature and the target text feature vector used to represent the target text feature can be fused to obtain a scene description of the video clip: Zhang San is slowly turning his head to watch the cat climbing a tree.

[0210] In one possible design, after the text fusion model outputs a situational description about a character, it can also be compared with the cast and crew list of the video to be processed to determine the real name, and then the real name is used to replace the character code in the text fusion model situation description to output a complete situation description.

[0211] For example, after the text fusion model outputs a scenario description about Zhang San, it can be compared with the cast and crew list of the video to be processed to determine that the real person's name is Li Si, and then Li Si is used to replace the character code Zhang San in the scenario description of the text fusion model, and the complete scenario description is output as: Zhang Si turns his head to look at the cat climbing a tree.

[0212] It should be understood that if the video to be processed includes only one video clip, the scene description of the video clip is also the scene description of the video to be processed; if the video to be processed includes at least two video clips, the scene descriptions of different video clips can be obtained sequentially through the text fusion model, and the same characters in different shot clips can be represented by the same character code, so as to obtain a complete scene description of the video to be processed.

[0213] The method for generating scene descriptions of video clips provided in this embodiment can obtain a video to be processed, perform physical lens segmentation on it, and obtain at least two video clips to be processed. The face images in these video clips are clustered to obtain at least one cluster center. The visual features of each cluster center and multiple sampling frames in the video clip are respectively obtained, and these visual features are mapped to text features to obtain target text features. Next, the prompt text input by the user is obtained, and its prompt text features are extracted. Then, the prompt text features are fused with the target text features corresponding to each visual feature, and finally the scene description of the video clip is obtained. Based on the prompt text input by the user, the text fusion model can be guided to output a scene description that better meets the user's needs and expectations, thereby reducing the work of manually summarizing and generalizing the video plot, shortening the time required to generate the scene description, and improving production efficiency.

[0214] The specific implementation scheme of the model training method provided by the present invention is introduced below.

[0215] Fig.10 is a schematic diagram of the seventh embodiment of the present disclosure, referring to Fig.10 The model training method shown in the figure specifically comprises the following steps:

[0216] S1001. Sample images in a video clip to obtain multiple sampling frames, and acquire visual features corresponding to each sampling frame.

[0217] In this step, the sampling interval of the sampling frames can be set for the video clip, so as to determine the number and position of the sampling frames and obtain multiple sampling frames. The visual features corresponding to each sampling frame are obtained through the visual self-attention network (Vit).

[0218] S1002: Extracting conversation data from a video clip to obtain a plurality of text features corresponding to the video clip.

[0219] In this step, the conversation data in the video clip may be extracted, and text features may be extracted from the conversation data to obtain a plurality of text features corresponding to the video clip.

[0220] For example, if the dialogue data is text content displayed in the sampling frame, the text content displayed in the sampling frame can be converted into text data through optical character recognition (OCR) technology, and multiple text features can be obtained through a natural language processing algorithm.

[0221] Exemplarily, if the conversation data is audio data in a sampling frame, the audio data can be converted into text data through speech recognition technology, and multiple text features can be obtained through a natural language processing algorithm.

[0222] S1003: Obtain multiple training samples according to the visual features corresponding to each sampling frame and the multiple text features, each training sample including a visual feature and a text feature corresponding to the calibrated visual feature.

[0223] In the process of model training, the training samples used for training are crucial. The richness of the training samples directly affects the accuracy and generalization ability of the model output. Therefore, the multiple training samples can be obtained from different types of video clips to ensure that the training samples cover various scenes and content.

[0224] S1004. Train the multi-layer perceptron model according to the multiple training samples to obtain a feature mapping model.

[0225] A multilayer perceptron (MLP) is a feedforward neural network with multiple layers of neurons. It consists of an input layer, at least one hidden layer, and an output layer, with neurons in each layer fully connected to neurons in the previous layer. MLP transforms inputs through nonlinear activation functions, enabling it to learn and approximate complex functional relationships. During training, MLP typically uses a backpropagation algorithm to adjust weights to optimize prediction accuracy.

[0226] Specifically, after generating training samples, these samples can be divided into training sets and test sets. The division ratio can be flexibly selected, for example, 50% of the data set can be used for training and 50% for testing, or 70% of the data can be used as a training set and 30% of the data can be used as a test set. There is no strict restriction on the specific division ratio.

[0227] Furthermore, based on the training set, the back propagation algorithm can be used to iteratively optimize the parameters of the multilayer perceptron model by minimizing the error between the model prediction value and the actual value to obtain the initial feature mapping model. Then, the initial feature mapping model is evaluated and adjusted based on the test set to finally obtain the optimized feature mapping model.

[0228] The model training method provided in this embodiment can obtain multiple sampling frames in a video clip and obtain the visual features corresponding to each sampling frame; by extracting the dialogue data in the video clip, multiple text features corresponding to the video clip can be obtained. Then, multiple training samples can be obtained based on the visual features corresponding to each sampling frame and the multiple text features, and the multi-layer perceptron model can be trained to obtain a feature mapping model. By combining visual features and text features, the model can understand the video content more comprehensively and improve the richness and accuracy of feature representation. This can further improve the accuracy and efficiency of generating scene descriptions of video clips.

[0229] Fig.11 is a schematic diagram of the eighth embodiment of the present disclosure, referring to Fig.11 The model training method shown in the figure specifically comprises the following steps:

[0230] S1101. Obtain multiple scene descriptions for multiple video clips.

[0231] Optionally, artificial intelligence (AI) can be used to analyze and process video content, extract key scenes and events in each video clip, and generate detailed scenario descriptions. These scenario descriptions not only include descriptions of visual elements, but can also be combined with dialogue data in the video to provide a more comprehensive and richer understanding of the scene.

[0232] S1102: For each video segment, extract features of the prompt text input by the user for the video segment to obtain at least one prompt text feature.

[0233] In this step, a natural language processing algorithm may be used to extract features of the prompt text input by the user for the video clip to obtain at least one prompt text feature.

[0234] S1103: Extracting conversation data of the video segment, and acquiring at least one target text feature corresponding to the video segment based on the conversation data.

[0235] The specific implementation of this step is the same as the implementation of step S1002 in the aforementioned embodiment, and will not be repeated here.

[0236] S1104: Acquire multiple training samples according to the scene description corresponding to each video clip, at least one prompt text feature, and at least one target text feature.

[0237] Similarly, during the training process of the model, the multiple training samples obtained may include those obtained from different types of video clips, and each training sample includes at least one prompt text feature, at least one target text feature, and a scenario description corresponding to a video clip.

[0238] S1105. Train the large language model based on multiple training samples to obtain a text fusion model.

[0239] Large Language Model (LLM) is a natural language processing model based on deep learning. It can understand and generate natural language text by pre-training and fine-tuning on large-scale text data. It is widely used in text generation, machine translation and other fields. For example, a large language model can be an artificial intelligence model based on deep learning technology.

[0240] Similarly, after obtaining multiple training samples, these samples can be divided into training sets and test sets. Based on the training set, an optimizer is selected to iteratively optimize the large language model to obtain an initial text fusion model. Then, the initial text fusion model is evaluated and adjusted based on the test set to finally obtain an optimized text fusion model.

[0241] Optionally, during the iterative optimization of the large language model based on the training set, data augmentation techniques can be applied to increase the diversity and coverage of samples. By generating variant samples, performing synonym replacement or random insertion, the model's robustness to various language variants can be improved, thereby optimizing the model's performance.

[0242] The model training method provided in this embodiment can obtain multiple scene descriptions of multiple video clips, obtain at least one prompt text feature for each video clip, and obtain at least one target text feature corresponding to the video clip based on the dialogue data of the video clip. Then, based on the scene description corresponding to each video clip, at least one prompt text feature and at least one target text feature, multiple training samples can be obtained to train the large language model to obtain a text fusion model. By training the model with the scene description corresponding to each video clip, at least one prompt text feature and at least one target text feature, the model's ability to understand and generate video content can be significantly improved. By combining the multi-dimensional information of the video clip, the model can better capture the semantics and context of the video content, thereby improving the performance of the model in generating scene descriptions.

[0243] Fig.12 is a schematic diagram of the ninth embodiment of the present disclosure, referring to Fig.12 The model training method shown in the figure specifically comprises the following steps:

[0244] S1201. Obtain multiple initial videos.

[0245] Optionally, multiple initial videos can be generated by AI.

[0246] In another possible design, multiple initial videos can be obtained through a public video resource library or a public video classification platform.

[0247] For each initial video, at least one physical shot switching position may be included.

[0248] S1202: Slice each physical shot switching position in each initial video to obtain multiple target videos corresponding to each initial video.

[0249] Optionally, the location of physical lens switching can be determined through transition points between different lenses in the initial video, such as scene changes, character changes, sudden changes in visual features, color histogram changes, etc., to complete the segmentation of the initial video and obtain multiple target videos corresponding to each initial video.

[0250] S1203. According to the multiple initial videos and the multiple target videos corresponding to each initial video, a convolutional neural network model is trained to obtain a video segmentation model.

[0251] After obtaining multiple initial videos and multiple target videos corresponding to each initial video, multiple training samples can be constructed, and the training samples can be divided into training sets and test sets. Based on the training set, the convolutional neural network model is iteratively optimized to obtain an initial video segmentation model, and the initial video segmentation model is evaluated and adjusted based on the test set, so as to finally obtain an optimized video segmentation model. Each training sample includes an initial video and at least one target video corresponding to the initial video.

[0252] Convolutional Neural Network (CNN) is a deep learning model specifically designed to process data with a grid structure (such as images). It is widely used in tasks such as image classification, object detection, and image segmentation.

[0253] In one possible design, during the iterative optimization of the convolutional neural network model based on the training set, the loss between the prediction and the actual target (such as cross entropy loss or mean square error) can be calculated, and the gradient information can be passed back to the network layer through the back propagation algorithm to update the weights and biases to reduce the loss, so that the model can be continuously improved during the training process and gradually improve its ability to fit the training data.

[0254] The model training method provided in this embodiment can obtain multiple initial videos, segment each physical lens switching position in each initial video, and obtain multiple target videos corresponding to each initial video; according to the multiple initial videos and the multiple target videos corresponding to each initial video, the convolutional neural network model is trained to obtain a video segmentation model. By training the convolutional neural network model with a large amount of training data, a high-performance video segmentation model can be obtained, which can accurately identify the physical lens switching position in the video to be processed, thereby realizing automatic video segmentation and improving the efficiency of generating video clip scene descriptions.

[0255] Fig.13 is a schematic diagram of the tenth embodiment of the present disclosure, referring to Fig.13 The video clip scene description generating device 1300 shown includes the following units:

[0256] The first processing unit 1301 is used to cluster the face images in the video clip to be processed to obtain at least one cluster center;

[0257] The second processing unit 1302 is used to obtain visual features of at least one cluster center and a plurality of sample frames, respectively, where the plurality of sample frames are image frames extracted from the video clips and representing video content;

[0258] The third processing unit 1303 is used to perform text feature mapping processing on each visual feature to obtain a target text feature corresponding to each visual feature;

[0259] The fourth processing unit 1304 is used to fuse the prompt text features of the prompt text input by the user and the target text features corresponding to each visual feature to obtain a scene description of the video clip.

[0260] In a possible design, the third processing unit 1303 includes:

[0261] The first processing module is used to use a feature mapping model to map each visual feature to a text feature to obtain a target text feature corresponding to each visual feature; wherein the feature mapping model is a model for mapping visual features to text features obtained by training a multi-layer perceptron model.

[0262] In one possible design, the fourth processing unit 1304 includes:

[0263] The first computing module is used to use a text fusion model to fuse the prompt text features of the prompt text input by the user and the target text features corresponding to each visual feature to obtain a scene description of the video clip; wherein the text fusion model is a model for text feature fusion obtained by training a large language model.

[0264] In a possible design, the video clip scene description generating device further includes:

[0265] The fifth processing unit 1305 is used to perform physical shot segmentation on the video to be processed to obtain at least two video segments to be processed, and the segmentation points between the video segments are locations for physical shot switching.

[0266] In one possible design, the fifth processing unit 1305 includes:

[0267] The first segmentation module is used to perform physical lens segmentation on the video to be processed according to the video segmentation model to obtain at least two video clips to be processed. The video segmentation model is a model that segments the video based on the physical lens switching position obtained by training the convolutional neural network model.

[0268] In a possible design, the video clip scene description generating device further includes:

[0269] The sixth processing unit 1306 is configured to set the same identifier for at least two cluster centers whose facial feature similarity is higher than a preset value among the cluster centers corresponding to all the video clips.

[0270] In one possible design, the first processing unit 1301 includes:

[0271] A first processing module, used to extract multiple face images from the video clip using a face detection algorithm;

[0272] A second processing module, configured to cluster the multiple face images based on the face features in each face image to obtain at least one cluster;

[0273] The third processing module is used to obtain at least one cluster center based on each cluster.

[0274] In a possible design, the third processing module includes:

[0275] The first processing submodule is used to obtain the initial cluster center of each cluster;

[0276] The second processing submodule is used to calculate, for each cluster, the similarity between each face image in the cluster and the initial cluster center of the cluster;

[0277] The third processing submodule is used to select at least one target face image with the highest similarity to the initial cluster center from the cluster clusters, and determine the cluster center of the cluster cluster based on the at least one target face image.

[0278] In a possible design, the first processing submodule is specifically used for:

[0279] If the number of target face images is one, the target face image is used as the cluster center of the cluster;

[0280] If the number of target face images is greater than one, the means of at least two target face images are used as cluster centers of clusters.

[0281] Fig.14 is a schematic diagram of the eleventh embodiment of the present disclosure, referring to Fig.14 The model training device 1400 shown includes the following units:

[0282] The sampling unit 1401 is used to sample the images in the video clip to obtain multiple sampling frames, and obtain the visual features corresponding to each sampling frame;

[0283] An extraction unit 1402 is used to extract the conversation data in the video segment to obtain a plurality of text features corresponding to the video segment;

[0284] The processing unit 1403 is used to obtain multiple training samples according to the visual features corresponding to each sampling frame and the multiple text features, each training sample includes a visual feature and a text feature corresponding to the calibrated visual feature;

[0285] The training unit 1404 is used to train the multi-layer perceptron model according to multiple training samples to obtain a feature mapping model.

[0286] Fig.15 is a schematic diagram of the twelfth embodiment of the present disclosure, referring to Fig.15 The model training device 1500 shown may include the following units:

[0287] An acquisition unit 1501 is used to acquire multiple scene descriptions of multiple video clips;

[0288] The first extraction unit 1502 is used to extract features of the prompt text input by the user for each video segment to obtain at least one prompt text feature;

[0289] The second extraction unit 1503 is used to extract the dialogue data of the video segment, and obtain at least one target text feature corresponding to the video segment according to the dialogue data;

[0290] The processing unit 1504 is used to obtain a plurality of training samples according to the scene description corresponding to each video segment, at least one prompt text feature and at least one target text feature, wherein each training sample includes at least one prompt text feature corresponding to a video segment, at least one target text feature and the scene description;

[0291] The training unit 1505 is used to train the large language model according to multiple training samples to obtain a text fusion model.

[0292] Fig.16 is a schematic diagram of the thirteenth embodiment of the present disclosure, referring to Fig.16 The model training device 1600 shown may include the following units:

[0293] An acquisition unit 1601 is used to acquire multiple initial videos;

[0294] A segmentation unit 1602 is used to segment each physical shot switching position in each initial video to obtain multiple target videos corresponding to each initial video;

[0295] The training unit 1603 is used to train the convolutional neural network model according to multiple initial videos and multiple target videos corresponding to each initial video to obtain a video segmentation model.

[0296] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0297] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0298] Fig.17 A schematic block diagram of an example electronic device 1700 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0299] like Fig.17 As shown, the electronic device 1700 includes a computing unit 1701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1702 or a computer program loaded from a storage unit 1708 into a random access memory (RAM) 1703. In the RAM 1703, various programs and data required for the operation of the electronic device 1700 can also be stored. The computing unit 1701, the ROM 1702, and the RAM 1703 are connected to each other via a bus 1704. An input / output (I / O) interface 1705 is also connected to the bus 1704.

[0300] Multiple components in the electronic device 1700 are connected to the I / O interface 1705, including: an input unit 1706, such as a keyboard, a mouse, etc.; an output unit 1707, such as various types of displays, speakers, etc.; a storage unit 1708, such as a disk, an optical disk, etc.; and a communication unit 1709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1709 allows the electronic device 1700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0301] The computing unit 1701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1701 performs the various methods and processes described above, such as the method video clip scene description generation method and the model training method. For example, in some embodiments, the method video clip scene description generation method and the model training method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1700 via the ROM 1702 and / or the communication unit 1709. When the computer program is loaded into the RAM 803 and executed by the computing unit 1701, one or more steps of the method video clip scene description generation method and the model training method described above may be executed. Alternatively, in other embodiments, the computing unit 1701 may be configured to execute the method video clip scene description generation method and the model training method in any other appropriate manner (for example, by means of firmware).

[0302] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0303] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0304] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0305] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0306] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0307] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0308] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0309] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for generating a scene description of a video clip, comprising: Clustering the face images in the video clip to be processed to obtain at least one cluster center; Respectively acquiring visual features of the at least one cluster center and a plurality of sampling frames, where the plurality of sampling frames are image frames representing video content extracted from the video clip; Mapping each visual feature to a text feature to obtain a target text feature corresponding to each visual feature; The prompt text features of the prompt text input by the user and the target text features corresponding to each visual feature are fused to obtain a scene description of the video clip.

2. The method according to claim 1, wherein: The mapping process of each visual feature to a text feature to obtain a target text feature corresponding to each visual feature includes: A feature mapping model is used to map text features to each visual feature to obtain target text features corresponding to each visual feature; wherein the feature mapping model is a model for mapping visual features and text features obtained by training a multi-layer perceptron model.

3. The method according to claim 1, wherein: The prompt text feature of the prompt text input by the user and the target text feature corresponding to each visual feature are fused to obtain the scene description of the video clip, including: A text fusion model is used to fuse the prompt text features of the prompt text input by the user and the target text features corresponding to each visual feature to obtain the scene description of the video clip; wherein the text fusion model is a model for text feature fusion obtained by training a large language model.

4. The method according to any one of claims 1 to 3, wherein: Before clustering the face images in the video clip to be processed to obtain at least one cluster center, the method further includes: The video to be processed is physically divided into shots to obtain at least two video segments to be processed, and the dividing points between the video segments are the locations for physical shot switching.

5. The method according to claim 4, wherein: The physical shot segmentation of the video to be processed to obtain at least two video segments to be processed includes: According to the video segmentation model, the video to be processed is physically segmented to obtain the at least two video clips to be processed, and the video segmentation model is a model for segmenting the video based on the physical lens switching position obtained by training a convolutional neural network model.

6. The method according to claim 4, wherein: The method further comprises: Among the cluster centers corresponding to all the video clips, at least two cluster centers whose facial feature similarity is higher than a preset value are set with the same identifier.

7. The method according to any one of claims 1 to 3, wherein: The step of clustering the face images in the video clip to be processed to obtain at least one cluster center includes: Extracting multiple face images from the video clip using a face detection algorithm; Clustering the multiple face images based on the face features in each face image to obtain at least one cluster; Based on each cluster, the at least one cluster center is obtained.

8. The method according to claim 7, wherein: The acquiring the at least one cluster center based on each cluster includes: Get the initial cluster center of each cluster; For each cluster, respectively calculating the similarity between each face image in the cluster and the initial cluster center of the cluster; At least one target face image having the highest similarity to an initial cluster center is selected from the clusters, and the cluster center of the cluster is determined based on the at least one target face image.

9. The method according to claim 8, wherein: The step of determining the cluster center of the cluster cluster according to the at least one target face image comprises: If the number of target face images is one, the target face image is used as the cluster center of the cluster; If the number of target face images is greater than one, the mean of at least two target face images is used as the cluster center of the cluster.

10. A model training method, comprising: Sampling images in the video clip to obtain multiple sampling frames, and obtaining visual features corresponding to each sampling frame; Extracting the conversation data in the video clip to obtain a plurality of text features corresponding to the video clip; Obtaining a plurality of training samples according to the visual features corresponding to each sampling frame and the plurality of text features, each training sample comprising a visual feature and a calibrated text feature corresponding to the visual feature; The multi-layer perceptron model is trained according to the multiple training samples to obtain a feature mapping model.

11. A model training method, comprising: Obtain multiple scene descriptions for multiple video clips; For each video clip, extracting features of a prompt text input by a user for the video clip to obtain at least one prompt text feature; Extracting conversation data of the video clip, and acquiring at least one target text feature corresponding to the video clip based on the conversation data; According to the scene description corresponding to each video clip, at least one prompt text feature and at least one target text feature, a plurality of training samples are obtained, each training sample including at least one prompt text feature corresponding to a video clip, at least one target text feature and a scene description; The large language model is trained according to the multiple training samples to obtain a text fusion model.

12. A model training method, comprising: Get multiple initial videos; Slice each physical shot switching position in each initial video to obtain multiple target videos corresponding to each initial video; According to the multiple initial videos and the multiple target videos corresponding to each initial video, a convolutional neural network model is trained to obtain a video segmentation model.

13. A device for generating a scene description of a video clip, comprising: A first processing unit, configured to cluster the face images in the video clip to be processed to obtain at least one cluster center; a second processing unit, configured to respectively obtain visual features of the at least one cluster center and a plurality of sampling frames, wherein the plurality of sampling frames are image frames extracted from the video clip and representing video content; A third processing unit is used to perform text feature mapping processing on each visual feature to obtain a target text feature corresponding to each visual feature; The fourth processing unit is used to fuse the prompt text features of the prompt text input by the user and the target text features corresponding to each visual feature to obtain the scene description of the video clip.

14. The device according to claim 13, wherein: The third processing unit comprises: The first processing module is used to use a feature mapping model to map each visual feature to a text feature to obtain a target text feature corresponding to each visual feature; wherein the feature mapping model is a model for mapping visual features to text features obtained by training a multi-layer perceptron model.

15. The device according to claim 13, wherein: The fourth processing unit comprises: The first computing module is used to use a text fusion model to fuse the prompt text features of the prompt text input by the user and the target text features corresponding to each visual feature to obtain the scene description of the video clip; wherein the text fusion model is a model for text feature fusion obtained by training a large language model.

16. The device according to any one of claims 13 to 15, wherein: The device also includes: The fifth processing unit is used to perform physical shot segmentation on the video to be processed to obtain at least two video segments to be processed, and the segmentation points between the video segments are locations for physical shot switching.

17. The device according to claim 16, wherein: The fifth processing unit comprises: The first segmentation module is used to perform physical shot segmentation on the video to be processed according to a video segmentation model to obtain the at least two video clips to be processed. The video segmentation model is a model that segments the video based on the physical shot switching position obtained by training a convolutional neural network model.

18. The device according to claim 16, wherein: The device also includes: The sixth processing unit is used to set the same identifier for at least two cluster centers whose facial feature similarity is higher than a preset value among the cluster centers corresponding to all the video clips.

19. The device according to any one of claims 13 to 15, wherein: The first processing unit comprises: A first processing module, configured to extract a plurality of face images from the video clip using a face detection algorithm; A second processing module, configured to cluster the plurality of face images based on the face features in each face image to obtain at least one cluster; The third processing module is used to obtain the at least one cluster center based on each cluster.

20. The device according to claim 19, wherein The third processing module comprises: The first processing submodule is used to obtain the initial cluster center of each cluster; A second processing submodule is used to calculate, for each cluster, the similarity between each face image in the cluster and the initial cluster center of the cluster; The third processing submodule is used to select at least one target face image with the highest similarity to the initial cluster center from the cluster clusters, and determine the cluster center of the cluster cluster based on the at least one target face image.

21. The device according to claim 20, wherein: The first processing submodule is specifically used to: If the number of target face images is one, the target face image is used as the cluster center of the cluster; If the number of target face images is greater than one, the mean of at least two target face images is used as the cluster center of the cluster.

22. A model training device, comprising: A sampling unit, used to sample images in the video clip to obtain multiple sampling frames, and obtain visual features corresponding to each sampling frame; An extraction unit, used to extract the conversation data in the video clip to obtain a plurality of text features corresponding to the video clip; A processing unit, configured to obtain a plurality of training samples according to the visual features corresponding to each sampling frame and the plurality of text features, wherein each training sample includes a visual feature and a calibrated text feature corresponding to the visual feature; The training unit is used to train the multi-layer perceptron model according to the multiple training samples to obtain a feature mapping model.

23. A model training device, comprising: An acquisition unit, used for acquiring multiple scene descriptions of multiple video clips; A first extraction unit is used to extract features of the prompt text input by the user for each video clip to obtain at least one prompt text feature; a second extraction unit, configured to extract the conversation data of the video segment, and obtain at least one target text feature corresponding to the video segment according to the conversation data; A processing unit, configured to obtain a plurality of training samples according to a scene description corresponding to each video clip, at least one prompt text feature, and at least one target text feature, wherein each training sample includes at least one prompt text feature corresponding to a video clip, at least one target text feature, and a scene description; The training unit is used to train the large language model according to the multiple training samples to obtain a text fusion model.

24. A model training device, comprising: An acquisition unit, used for acquiring a plurality of initial videos; A segmentation unit, used for segmenting each physical shot switching position in each initial video to obtain multiple target videos corresponding to each initial video; A training unit is used to train a convolutional neural network model according to the multiple initial videos and the multiple target videos corresponding to each initial video to obtain a video segmentation model.

25. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 12.

26. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-12.

27. A computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 12.

Images