Video creation method and system

Abstract

The embodiment of the invention discloses a video creation method and system. The method comprises the steps of obtaining a material search request sent by a request object; the material search request is forwarded to a server, so that the server performs video material search according to the material search request to obtain a video material search result, and if copyright information of a first video material corresponding to the video material search result is first copyright information, visual data corresponding to the video material search result is returned to the client; and performing local video material search according to the visual data returned by the server to obtain a second video material, and displaying the second video material to the request object to enable the request object to perform video creation according to the second video material. According to the scheme provided by the embodiment of the invention, the request object can be helped to create a high-quality video, the user experience is improved, in addition, through an end-cloud cooperation mode, the video material is quickly retrieved, and the material retrieval speed and accuracy in the video creation process are effectively improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of Internet technology, and specifically to a video creation method and system. Background Art

[0002] With the rise of short video platforms, more and more users are participating in video production. However, the videos created by users are of low quality, unable to attract other users to watch, and are prone to infringement issues. Therefore, how to quickly create high-quality videos has become a problem that users urgently need to solve. Summary of the invention

[0003] In view of the above problems, the present application is proposed to provide a video creation method, system, computing device, computer storage medium and computer program product that overcome the above problems or at least partially solve the above problems.

[0004] According to one aspect of an embodiment of the present application, a video creation method is provided, the method is applied to a client, and the method includes:

[0005] Get the material search request sent by the request object;

[0006] forwarding the material search request to the server, so that the server searches for video materials according to the material search request, obtains a video material search result, and returns visual data corresponding to the video material search result to the client if the copyright information of the first video material corresponding to the video material search result is the first copyright information;

[0007] A local video material search is performed based on the visual data returned by the server to obtain a second video material, and the second video material is displayed to the requesting object so that the requesting object can create a video based on the second video material.

[0008] Furthermore, the method further includes: when the server determines that the copyright information of the first video material corresponding to the video material search result is the second copyright information, the server returns the first video material to the client;

[0009] The first video material returned by the server is received, and the first video material is displayed to the requesting object, so that the requesting object creates a video according to the first video material.

[0010] Furthermore, the method further includes: if the video material includes a video, extracting a video frame from the video material, and performing vectorization processing on the extracted video frame using a pre-trained multimodal vector model to obtain an image feature vector; and / or

[0011] If the video material contains images, the pre-trained multimodal vector model is used to vectorize the images to obtain image feature vectors. The multimodal vector model is trained based on image-text pairs.

[0012] Among them, the server associates and stores all or part of the following information: video material, video material identification, extracted video frames, image feature vectors, appearance timestamp of the video frame corresponding to the image feature vector in the video, and copyright information; the client locally associates and stores all or part of the following information: video material, video material identification, extracted video frames, and image feature vectors.

[0013] Furthermore, the material search request carries a search keyword;

[0014] The server searches for video materials according to the material search request, and obtains the video material search result further including:

[0015] The server uses a pre-trained multimodal vector model to vectorize the search keyword to obtain a search keyword feature vector;

[0016] Searching for video material based on the search keyword feature vector to obtain a matching first image feature vector, a video material identifier, and an appearance timestamp of a video frame corresponding to the image feature vector in the video; or; searching for video material based on the search keyword feature vector to obtain a matching first image and video material identifier;

[0017] The visual data corresponding to the video material search result returned to the client further includes:

[0018] The first image feature vector is returned to the client or the first image corresponding to the first image feature vector is returned to the client.

[0019] Furthermore, searching for local video material according to the visual data returned by the server to obtain the second video material further includes:

[0020] Calculate the similarity between the first image feature vector and the image feature vector stored locally on the client, and select a second image feature vector matching the first image feature vector according to the similarity;

[0021] A second video material corresponding to the second image feature vector is determined.

[0022] Furthermore, searching for local video material according to the visual data returned by the server to obtain the second video material further includes:

[0023] Vectorize the first image returned by the server using a pre-trained multimodal vector model to obtain a first image feature vector;

[0024] Calculate the similarity between the first image feature vector and the image feature vector stored locally on the client, and select a second image feature vector matching the first image feature vector according to the similarity;

[0025] A second video material corresponding to the second image feature vector is determined.

[0026] According to another aspect of the embodiment of the present application, a video creation system is provided, the system comprising a client and a server, wherein the client comprises: an acquisition module, a forwarding module, a search module, and a display module;

[0027] An acquisition module, adapted to acquire a material search request sent by a request object;

[0028] A forwarding module, adapted to forward the material search request to the server;

[0029] The server is adapted to perform a video material search according to the material search request, obtain a video material search result, and return visual data corresponding to the video material search result to the client if the copyright information of the first video material corresponding to the video material search result is the first copyright information;

[0030] A search module, adapted to search for local video material according to the visual data returned by the server to obtain a second video material;

[0031] The display module is suitable for displaying the second video material to the requesting object so that the requesting object can create a video based on the second video material.

[0032] According to another aspect of the embodiment of the present application, there is provided a computing device, including: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus;

[0033] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute operations corresponding to the above-mentioned video creation method.

[0034] According to another aspect of the embodiment of the present application, a computer storage medium is provided, in which at least one executable instruction is stored, and the executable instruction enables a processor to perform operations corresponding to the above-mentioned video creation method.

[0035] According to another aspect of the embodiments of the present application, a computer program product is provided, comprising at least one executable instruction, wherein the executable instruction enables a processor to perform operations corresponding to the above-mentioned video creation method.

[0036] According to the video creation method and system provided in the embodiment of the present application, by performing vectorization processing on the image, the material search efficiency can be improved, and by performing copyright identification on the first video material corresponding to the video material search result, when the copyright information corresponding to the first video material is the first copyright information, the server only returns the visual data corresponding to the video material search result, and does not transmit the video file, thereby helping the request to avoid copyright risks and avoid legal disputes caused by the use of unauthorized materials. When the copyright information corresponding to the first video material is the second copyright information, the corresponding video material is returned to the client. Finally, the client can perform a local search based on the visual data returned by the server, search for the video material that is closest to the search intention of the requesting object, and display it to the requesting object so that it can use the searched video material to create a video, thereby helping the requesting object to create a high-quality video and improve the user experience. In addition, the present application realizes rapid retrieval of video materials through end-cloud collaboration, effectively improving the speed and accuracy of material retrieval during the video creation process.

[0037] The above description is only an overview of the technical solution of the embodiment of the present application. In order to more clearly understand the technical means of the embodiment of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiment of the present application more obvious and easy to understand, the specific implementation method of the embodiment of the present application is listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only used for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the embodiments of the present application. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0039] Figure 1 A signaling interaction diagram of a video creation method according to an embodiment of the present application is shown;

[0040] Figure 2 A signaling interaction diagram of a video creation method according to another embodiment of the present application is shown;

[0041] Figure 3 A structural block diagram of a video creation system according to an embodiment of the present application is shown;

[0042] Figure 4 A schematic diagram of the structure of a computing device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0043] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0044] First, the terms involved in one or more embodiments of the present application are explained.

[0045] End-to-cloud collaboration: refers to the collaboration between the client and the cloud server, which aims to provide more powerful computing power and resource support to realize various applications and functions. In this mode, the client can communicate with the cloud server and hand over some computing tasks, data processing, storage and other tasks to the cloud, thereby reducing the burden on the client, saving energy and resources, and being able to perform more complex tasks with limited hardware resources.

[0046] Material library: is a collection of resources that provides images, pictures and other resources used for video production for users to download or use directly.

[0047] Vectorization: Vectorization is the process of converting data or information into vectors so that they can be processed and analyzed in computers. Through vectorization, different types of data (such as text, images) can be uniformly represented as numerical vectors, making it easier for computers to perform mathematical operations and statistical analysis. In the field of machine learning and data analysis, vectorization is often used to convert raw data into feature vectors for model training or pattern recognition.

[0048] Video material search and matching: refers to the process of quickly finding original video or image resources related to the required content through keywords or specific screening conditions.

[0049] CLIP: CLIP (Contrastive Language-Image Pretraining) is an innovative deep learning model that achieves cross-modal information matching by jointly learning images and texts. It has excellent image and text understanding capabilities and can be used for a variety of tasks such as image classification, text description generation, and image generation. Its unique contrastive learning method enables it to show great potential and broad application prospects in the fields of computer vision and natural language processing.

[0050] Figure 1 A signaling interaction diagram of a video creation method according to an embodiment of the present application is shown. Figure 1 As shown, the method comprises the following steps:

[0051] Step S101: the client obtains a material search request sent by a request object, and forwards the material search request to a server.

[0052] The client can be an application on the terminal, such as a video creation APP. The request object can be any user who has video creation needs. When the user is creating a video, he may need to find video materials. At this time, the APP can provide the user with a search page. The search page contains a search box. The user can enter the corresponding search keyword in the search box to trigger the material search request. The client obtains the material search request sent by the request object and forwards the material search request to the server, that is, the cloud server, which first performs the actual search task.

[0053] In step S102, the server searches for video materials according to the material search request and obtains video material search results. If the copyright information of the first video material corresponding to the video material search results is the first copyright information, the server returns the visual data corresponding to the video material search results to the client; if the copyright information of the first video material corresponding to the video material search results is the second copyright information, the server returns the first video material to the client.

[0054] After receiving the material search request forwarded by the client, the server searches for video materials in the database within the server according to the material search request, and obtains video material search results through the search, wherein the video material search results vary depending on the information stored in the database within the server, for example, it can be an image or an image feature vector.

[0055] Generally, each video material has a corresponding copyright, and the copyright information of the video material stored in the server can be: the first copyright information or the second copyright information, wherein the first copyright information refers to the video material that has obtained the video playback authorization from the copyright holder, but has not obtained the adaptation authorization, that is, the user cannot create a video based on the video material. For such video material, the server can store the corresponding video material but cannot provide it to the user for creation; the second copyright information refers to the video material that has obtained both the video playback authorization and the adaptation authorization from the copyright holder. For such video material, the server can store the corresponding video material and can also provide it to the user for creation;

[0056] The copyright information of video materials is different, and the subsequent processing methods will be different. Therefore, the server can first query to determine the specific copyright information of the first video material corresponding to the video material search result. If the copyright information of the first video material corresponding to the video material search result is the first copyright information, the server returns the visual data corresponding to the video material search result to the client, where the visual data here can be the video material search result itself, or it can be an image corresponding to the video material search result; if the copyright information of the first video material corresponding to the video material search result is the second copyright information, the server returns the first video material to the client.

[0057] In step S103, the client receives the first video material returned by the server, and displays the first video material to the requesting object, so that the requesting object creates a video based on the first video material; or, searches for local video materials based on the visual data returned by the server to obtain the second video material, and displays the second video material to the requesting object, so that the requesting object creates a video based on the second video material.

[0058] The client performs different processing on different contents returned by the server. If the client receives the first video material from the server, the client can directly display the first video material to the request object. For example, the client can generate a display page, display the first video material on the display page, so that the request object can create a video based on the first video material.

[0059] If the client receives visual data from the server, the client needs to further search for video materials. Specifically, the client searches for local video materials based on the visual data returned by the server. Local video material search refers to searching locally on the client. The search is for video materials imported by the requesting object to the client. The second video material can be obtained through the search. Then, the client can display the second video material to the requesting object. For example, the client can generate a display page, display the second video material on the display page, so that the requesting object can create a video based on the second video material. In this embodiment, the video material can be a video and / or an image.

[0060] According to the video creation method provided by the embodiment of the present application, first, after the client receives the material search request of the request object, it forwards it to the server, and the server searches according to the material search request, and when the copyright information corresponding to the first video material corresponding to the video material search result is the first copyright information, the server only returns the visual data corresponding to the video material search result, and does not transmit the video file, thereby helping the request to avoid copyright risks and avoid legal disputes caused by the use of unauthorized materials. When the copyright information corresponding to the first video material is the second copyright information, the corresponding video material is returned to the client. Finally, the client can perform a local search based on the visual data returned by the server to search for the video material that is closest to the search intention of the request object, and display it to the request object (the video material displayed to the request object can be the video material obtained by the local search or the video material returned by the server), so that the request object can use the searched video material to create a video, thereby helping the request object to create high-quality videos and improve user experience. In addition, the present application realizes rapid retrieval of video materials through end-cloud collaboration, effectively improving the speed and accuracy of material retrieval during video creation.

[0061] Figure 2 A signaling interaction diagram of a video creation method according to another embodiment of the present application is shown. Figure 2 As shown, the method comprises the following steps:

[0062] In step S201, the server extracts video frames from the video in the server material library, and uses a pre-trained multimodal vector model to vectorize the extracted video frames to obtain image feature vectors; and / or, the server uses a pre-trained multimodal vector model to vectorize images in the material library to obtain image feature vectors, wherein the multimodal vector model is obtained based on image-text pair training, and the material library stores associated storage: video material, video material identification, and copyright information.

[0063] Specifically, a lot of video materials are imported into the server material library in batches, for example, videos and / or images. In order to facilitate the subsequent video material search, the video materials in the server material library need to be vectorized. Since the video contains many video frames, in order to improve the efficiency of video material search, some video frames can be extracted from the video, for example, extracting key frames or extracting a video frame every preset time, or extracting a video frame every preset number of video frames, and then inputting the extracted video frames into the pre-trained multimodal vector model deployed by the server, and the pre-trained multimodal vector model performs vectorization on the extracted video frames to obtain image feature vectors. Among them, the pre-trained multimodal vector model is obtained based on image-text pair training, for example, it can be an open source CLIP model, or it can be a model that fine-tunes the open source CLIP model using task-related image-text pairs.

[0064] In the case where the video material contains an image, the server inputs the image into the multimodal vector model, and the pre-trained multimodal vector model vectorizes the image in the material library to obtain the image feature vector. In order to facilitate the subsequent identification of whether the video material has copyright, the material library can store the following associated information: video material, video material identifier, and copyright information. The copyright information reflects the copyright status of the video material, where the copyright information can be the second copyright information or the first copyright information.

[0065] In step S202, the server associates and stores the image feature vector, the video material identifier, and the appearance timestamp of the video frame corresponding to the image feature vector in the video in a cloud vector database.

[0066] After vectorization, the server associates and stores the image feature vector, the video material identifier, and the appearance timestamp of the video frame corresponding to the image feature vector in the video in the cloud vector database. The server material library and the cloud vector database are associated through the video material identifier, wherein the appearance timestamp of the video frame corresponding to the image feature vector in the video refers to the time point when the video frame appears in the video frame. If the video material is an image, there is no need to store the appearance timestamp of the video frame corresponding to the image feature vector in the video. If the video material is a video, it is necessary to store the appearance timestamp of the video frame corresponding to the image feature vector in the video. The appearance timestamp of the video frame corresponding to the image feature vector in the video is stored here to facilitate the determination of the image corresponding to the image feature vector. In the case where the server material library does not store the image, it is convenient to extract the corresponding image from the corresponding video according to the appearance timestamp of the video frame corresponding to the image feature vector in the video.

[0067] In this embodiment, the server material library and the cloud vector database can also be merged into one database, for example, the image feature vector, the video material identifier, the appearance timestamp of the video frame corresponding to the image feature vector in the video, the video material, and the copyright information are all stored in the server material library. Alternatively, the server material library stores in association: the image feature vector, the appearance timestamp of the video frame corresponding to the image feature vector in the video, the video material, and the copyright information.

[0068] In addition, the server material library may also store the extracted video frames. In this case, the appearance timestamp of the video frame corresponding to the image feature vector in the video may no longer be stored.

[0069] In step S203, the client extracts video frames from the video in the local material library, and uses a pre-trained multimodal vector model to vectorize the extracted video frames to obtain image feature vectors; and / or the client uses a pre-trained multimodal vector model to vectorize the images in the material library to obtain image feature vectors, and the local material library associates and stores: video material, video material identifier.

[0070] In this embodiment, the client also pre-deploys a pre-trained multimodal vector model locally and indexes the client's local material library. The specific implementation of this step is similar to step S201 and will not be repeated here. In this step, the user can first batch import video materials into the local material library.

[0071] Step S204: the client associates and stores the image feature vector and the video material identifier in a local vector database.

[0072] In this embodiment, the client uses a local material library and a local vector database to store information, wherein the local material library stores the associated video material and video material identifier, and the local vector database stores the associated image feature vector and video material identifier. The local material library and the local vector database of the client are indexed by the associated video material identifier.

[0073] Step S205: the client obtains the material search request sent by the request object, and forwards the material search request to the server.

[0074] The client can be an application on the terminal, such as a video creation APP. The request object can be any user who has video creation needs. When the user is creating a video, he may need to find video materials. At this time, the APP can provide the user with a search page. The search page contains a search box. The user can enter the corresponding search keyword in the search box to trigger the material search request. The client obtains the material search request sent by the request object and forwards the material search request to the server, that is, the cloud server, which first performs the actual search task.

[0075] Step S206: The server uses a pre-trained multimodal vector model to vectorize the search keyword to obtain a search keyword feature vector.

[0076] Specifically, the material search request forwarded by the client carries the search keyword, so the server can extract the search keyword from the material search request, input the search keyword into the pre-trained multimodal vector model, and use the multimodal vector model to vectorize the search keyword to obtain the search keyword feature vector. Since the multimodal vector model is trained using image-text pairs, the model has learned the intrinsic connection between images and texts very well, and also enables image features and text features to be aligned in the same space, so that image feature vectors can be searched based on the search keyword feature vector in the future.

[0077] In step S207, the server queries the cloud vector database according to the search keyword feature vector, obtains the first image feature vector that matches the search keyword feature vector, the video material identifier, and the appearance timestamp of the video frame corresponding to the image feature vector in the video, and queries the server material library according to the video material identifier to determine the copyright information corresponding to the first video material corresponding to the video material identifier; if the copyright information is the second copyright information, the first video material is returned to the client; if the copyright information is the first copyright information, the video frame is extracted from the video corresponding to the video material identifier according to the appearance timestamp of the video frame corresponding to the image feature vector in the video, the extracted video frame is determined as the first image or the image corresponding to the first image feature vector is determined as the first image, and the first image is returned to the client.

[0078] After the server quantifies the search keyword, it can query the cloud vector database based on the search keyword feature vector, and match the search keyword feature vector with the image feature vector in the cloud vector database. For example, the similarity between the search keyword feature vector and the image feature vector stored in the cloud vector database can be calculated, and the first image feature vector that matches the search keyword feature vector can be obtained by filtering based on the similarity. For example, image feature vectors whose similarity is greater than or equal to a first preset similarity threshold can be filtered, and the filtered image feature vector can be used as the first image feature vector, or N image feature vectors with the highest similarity can be selected as the first image feature vector.

[0079] Usually, each video material has a corresponding copyright, and the copyright information of the video material stored in the server can be: the first copyright information or the second copyright information, wherein the first copyright information refers to the video material that has obtained the video playback authorization from the copyright holder, but has not obtained the adaptation authorization, that is, the user cannot create a video based on the video material. For such video material, the server can store the corresponding video material but cannot provide it to the user for creation; the second copyright information refers to the video material that has obtained both the video playback authorization and the adaptation authorization from the copyright holder. For such video material, the server can store the corresponding video material and can also provide it to the user for creation;

[0080] The copyright information of video materials is different, and the subsequent processing methods will be different. Therefore, after determining the matching first image feature vector, the cloud-based vector database can be queried to determine the associated video material identifier and the appearance timestamp of the video frame corresponding to the image feature vector in the video. Then, the server material library is queried based on the video material identifier to determine whether the copyright information associated with the video material identifier is the second copyright information or the first copyright information. If the copyright information is the second copyright information, the first video material is returned to the client.

[0081] In the case where the copyright information is the first copyright information, if the video material corresponding to the first image feature vector is a video, then the video frame is extracted from the video corresponding to the video material identifier according to the appearance timestamp of the video frame corresponding to the image feature vector in the video, and then the extracted video frame is determined as the first image, and the first image is returned to the client. If the video material corresponding to the first image feature vector is an image, the image corresponding to the first image feature vector can be directly determined as the first image, and the first image is returned to the client. In the case where the video material is an image, it is preferred to return the first image feature vector to the client to avoid directly returning the image corresponding to the first image feature vector to the client.

[0082] In an optional implementation, when the server material library stores the extracted video frame corresponding to the image feature vector, the extracted video frame corresponding to the image feature vector may be directly determined as the first image and returned to the client.

[0083] In the case where the server material library stores extracted video frames corresponding to image feature vectors, the image corresponding to the first image feature vector matching the search keyword feature vector may be directly used as the search result.

[0084] In an optional implementation, the server may also return the first image feature vector to the client, so that the client can directly perform a local material search based on the first image feature vector returned by the server, thereby improving the material search efficiency.

[0085] In step S208, the client receives the first video material returned by the server, and displays the first video material to the requesting object, so that the requesting object can create a video based on the first video material; alternatively, the client uses a pre-trained multimodal vector model to vectorize the first image returned by the server to obtain a first image feature vector; calculates the similarity between the first image feature vector and the image feature vector stored locally on the client, and selects a second image feature vector that matches the first image feature vector based on the similarity; determines the second video material corresponding to the second image feature vector, and displays the second video material to the requesting object, so that the requesting object can create a video based on the second video material.

[0086] The client performs different processing on different contents returned by the server. If the client receives the first video material from the server, the client can directly display the first video material to the request object. For example, the client can generate a display page, display the first video material on the display page, so that the request object can create a video based on the first video material.

[0087] If the client receives the first image from the server, the client needs to further search for video materials. Specifically, the client first inputs the first image into a pre-trained multimodal vector model deployed locally, and the multimodal vector model vectorizes the first image to obtain a first image feature vector. Then, the first image feature vector is used to search for local materials, wherein the local video material search refers to a search performed locally on the client, and the search is for video materials imported into the client by the request object. For example, the similarity between the first image feature vector and the image feature vector stored locally on the client can be calculated, and the second image feature vector matching the first image feature vector can be filtered according to the similarity. For example, image feature vectors whose similarity is greater than or equal to a second preset similarity threshold are filtered, and the filtered image feature vectors are filtered. as the second image feature vector, or, selecting M image feature vectors with the highest similarity as the second image feature vector, wherein the second preset similarity threshold may be the same as or different from the first preset similarity threshold, and the values ​​of M and N may be the same or different; determining the second video material corresponding to the second image feature vector, and displaying the second video material to the requesting object, so that the requesting object creates a video according to the second video material. For example, the local material library may be queried through the video material identifier corresponding to the second image feature vector to obtain the second video material associated with the video material identifier in the local material library. Then, the client may display the second video material to the requesting object. For example, the client may generate a display page, and display the second video material on the display page, so that the requesting object creates a video according to the second video material.

[0088] In an optional embodiment, if the client receives a first image feature vector from the server, the first image feature vector can be directly used to search for local materials. For example, the similarity between the first image feature vector and the image feature vector stored locally on the client is calculated, and a second image feature vector matching the first image feature vector is selected based on the similarity; and the second video material corresponding to the second image feature vector is determined.

[0089] According to the video creation method provided in the embodiment of the present application, by performing vectorization processing on the image, the material search efficiency can be improved, and by performing copyright identification on the first video material corresponding to the video material search result, when the copyright information corresponding to the first video material is the first copyright information, the server only returns the visual data corresponding to the video material search result, and does not transmit the video file, thereby helping the request to avoid copyright risks and avoid legal disputes caused by the use of unauthorized materials. When the copyright information corresponding to the first video material is the second copyright information, the corresponding video material is returned to the client. Finally, the client can perform a local search based on the visual data returned by the server, search for the video material that is closest to the search intention of the requesting object, and display it to the requesting object so that it can use the searched video material to create a video, thereby helping the requesting object to create a high-quality video and improve the user experience. In addition, the present application realizes rapid retrieval of video materials through end-cloud collaboration, effectively improving the speed and accuracy of material retrieval during the video creation process.

[0090] Figure 3 A structural block diagram of a video creation system according to an embodiment of the present application is shown. Figure 3 As shown, the system includes: a client 310 and a server 320, wherein the client includes: an acquisition module 311, a forwarding module 312, a search module 313, and a display module 314;

[0091] An acquisition module 311 is adapted to acquire a material search request sent by a request object;

[0092] A forwarding module 312, adapted to forward the material search request to a server;

[0093] The server 320 is adapted to perform a video material search according to the material search request, obtain a video material search result, and return visual data corresponding to the video material search result to the client if the copyright information of the first video material corresponding to the video material search result is the first copyright information;

[0094] A search module 313, adapted to search for local video material according to the visual data returned by the server to obtain a second video material;

[0095] The display module 314 is adapted to display the second video material to the requesting object, so that the requesting object can create a video based on the second video material.

[0096] Optionally, the server is further adapted to: if it is determined that the copyright information of the first video material corresponding to the video material search result is the second copyright information, then return the first video material to the client;

[0097] The display module is further adapted to: receive the first video material returned by the server, and display the first video material to the requesting object, so that the requesting object can create a video according to the first video material.

[0098] Optionally, the client further includes: a vectorization processing module, and the server or the vectorization processing module is further adapted to: if the video material includes a video, extract a video frame from the video material, and use a pre-trained multimodal vector model to vectorize the extracted video frame to obtain an image feature vector; and / or

[0099] If the video material contains images, the pre-trained multimodal vector model is used to vectorize the images to obtain image feature vectors. The multimodal vector model is trained based on image-text pairs.

[0100] Among them, the server associates and stores all or part of the following information: video material, video material identification, extracted video frames, image feature vectors, appearance timestamp of the video frame corresponding to the image feature vector in the video, and copyright information; the client locally associates and stores all or part of the following information: video material, video material identification, extracted video frames, and image feature vectors.

[0101] Optionally, the material search request carries a search keyword;

[0102] The server is further adapted to: vectorize the search keyword using a pre-trained multimodal vector model to obtain a search keyword feature vector;

[0103] Searching for video material based on the search keyword feature vector to obtain a matching first image feature vector, a video material identifier, and an appearance timestamp of a video frame corresponding to the image feature vector in the video; or; searching for video material based on the search keyword feature vector to obtain a matching first image and video material identifier;

[0104] The first image feature vector is returned to the client or the first image corresponding to the first image feature vector is returned to the client.

[0105] Optionally, the search module is further adapted to: calculate the similarity between the first image feature vector and the image feature vector locally stored in the client, and select the second image feature vector matching the first image feature vector according to the similarity;

[0106] A second video material corresponding to the second image feature vector is determined.

[0107] Optionally, the search module is further adapted to: vectorize the first image returned by the server using a pre-trained multimodal vector model to obtain a first image feature vector;

[0108] Calculate the similarity between the first image feature vector and the image feature vector stored locally on the client, and select a second image feature vector matching the first image feature vector according to the similarity;

[0109] A second video material corresponding to the second image feature vector is determined.

[0110] The description of each module above refers to the corresponding description in the method embodiment and will not be repeated here.

[0111] According to the video creation system provided in the embodiment of the present application, the material search efficiency can be improved by vectorizing the image, and the copyright identification is performed on the first video material corresponding to the video material search result. When the first video material does not have the second copyright information, the server only returns the visual data corresponding to the video material search result without transmitting the video file. This can help the requester avoid copyright risks and avoid legal disputes caused by the use of unauthorized materials. In the case of the second copyright information, the corresponding video material is returned to the client. Finally, the client can perform a local search based on the visual data returned by the server to search for the video material that is closest to the search intention of the requesting object, and display it to the requesting object so that it can use the searched video material to create a video. This can help the requesting object create high-quality videos and improve user experience. In addition, the present application realizes rapid retrieval of video materials through end-cloud collaboration, effectively improving the speed and accuracy of material retrieval during video creation.

[0112] An embodiment of the present application provides a non-volatile computer storage medium, which stores at least one executable instruction or computer program, and the executable instruction or computer program can enable a processor to execute operations corresponding to the video creation method in any of the above method embodiments.

[0113] An embodiment of the present application provides a computer program product, which includes at least one executable instruction or computer program, and the executable instruction or computer program can enable a processor to execute operations corresponding to the video creation method in any of the above-mentioned method embodiments.

[0114] Figure 4 A schematic diagram of the structure of an embodiment of a computing device of the present application is shown. The specific embodiment of the present application does not limit the specific implementation of the computing device.

[0115] like Figure 4 As shown, the computing device may include: a processor (processor) 402 , a communications interface (Communications Interface) 404 , a memory (memory) 406 , and a communication bus 408 .

[0116] The processor 402, the communication interface 404, and the memory 406 communicate with each other via the communication bus 408. The communication interface 404 is used to communicate with other devices such as a client or other server network elements. The processor 402 is used to execute the program 410, which can specifically execute the relevant steps in the above-mentioned video creation method embodiment for a computing device.

[0117] Specifically, the program 410 may include program codes, which include computer operation instructions.

[0118] The processor 402 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the computing device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0119] The memory 406 is used to store the program 410. The memory 406 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0120] Program 410 can be specifically used to enable processor 402 to execute the video creation method in any of the above-mentioned method embodiments. The specific implementation of each step in program 410 can refer to the corresponding description in the corresponding steps and units in the above-mentioned video creation embodiment, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the above-mentioned method embodiment, which will not be repeated here.

[0121] The algorithm and display provided herein are not inherently related to any specific computer, virtual system or other equipment. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious to construct the structure required for this type of system. In addition, the present application embodiment is not directed to any specific programming language yet. It should be understood that various programming languages ​​can be utilized to realize the content of the present application embodiment described herein, and the above description of specific languages ​​is to disclose the best mode of implementation of the present application embodiment.

[0122] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0123] Similarly, it should be understood that in order to streamline the present disclosure and help understand one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the embodiments of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following intention: the claimed embodiments of the present application require more features than the features clearly stated in each claim. More specifically, as reflected in the claims below, the inventive aspects are less than all the features of the single embodiment disclosed above. Therefore, the claims following the specific embodiment are hereby explicitly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the embodiment of the present application.

[0124] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0125] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the embodiments of the present application and to form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0126] The various component embodiments of the embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all functions of some or all components according to the embodiments of the present application. The embodiments of the present application can also be implemented as a device or apparatus program (e.g., computer program and computer program product) for executing part or all of the methods described herein. Such a program implementing the embodiments of the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0127] It should be noted that the above embodiments illustrate rather than limit the embodiments of the present application, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets should not be constructed as a limitation to the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of multiple such elements. The embodiments of the present application may be implemented by means of hardware including several different elements and by means of appropriately programmed computers. In a unit claim that lists several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names.

Claims

1. A video creation method, the method being applied to a client, the method comprising: Get the material search request sent by the request object; forwarding the material search request to a server, so that the server searches for video materials according to the material search request to obtain a video material search result, and if the copyright information of the first video material corresponding to the video material search result is the first copyright information, returning visual data corresponding to the video material search result to the client; A local video material search is performed based on the visual data returned by the server to obtain a second video material, and the second video material is displayed to the requesting object so that the requesting object can create a video based on the second video material.

2. The method according to claim 1, wherein: The method further includes: when the server determines that the copyright information of the first video material corresponding to the video material search result is the second copyright information, the server returns the first video material to the client; Receive the first video material returned by the server, and display the first video material to the requesting object, so that the requesting object creates a video according to the first video material.

3. The method according to claim 1 or 2, wherein: The method further includes: if the video material includes a video, extracting a video frame from the video material, and performing vectorization processing on the extracted video frame using a pre-trained multimodal vector model to obtain an image feature vector; and / or If the video material includes an image, the image is vectorized using a pre-trained multimodal vector model to obtain an image feature vector, wherein the multimodal vector model is obtained based on image-text pair training; Among them, the server associates and stores all or part of the following information: video material, video material identification, extracted video frames, image feature vectors, appearance timestamp of the video frame corresponding to the image feature vector in the video, and copyright information; the client locally associates and stores all or part of the following information: video material, video material identification, extracted video frames, and image feature vectors.

4. The method according to claim 3, wherein: The material search request carries a search keyword; The server searches for video material according to the material search request, and obtains video material search results further comprising: The server performs vectorization processing on the search keyword using a pre-trained multimodal vector model to obtain a search keyword feature vector; Searching for video material according to the search keyword feature vector to obtain a matching first image feature vector, a video material identifier, and an appearance timestamp of a video frame corresponding to the image feature vector in the video; or; searching for video material according to the search keyword feature vector to obtain a matching first image and video material identifier; The returning of visual data corresponding to the video material search result to the client further comprises: The first image feature vector is returned to the client or the first image corresponding to the first image feature vector is returned to the client.

5. The method according to claim 4, wherein: The searching of the local video material according to the visual data returned by the server to obtain the second video material further comprises: Calculating the similarity between the first image feature vector and an image feature vector locally stored on the client, and selecting a second image feature vector matching the first image feature vector according to the similarity; Determine a second video material corresponding to the second image feature vector.

6. The method according to claim 4, wherein: The searching of the local video material according to the visual data returned by the server to obtain the second video material further comprises: Using a pre-trained multimodal vector model, vectorize the first image returned by the server to obtain a first image feature vector; Calculating the similarity between the first image feature vector and an image feature vector locally stored on the client, and selecting a second image feature vector matching the first image feature vector according to the similarity; Determine a second video material corresponding to the second image feature vector.

7. A video creation system, the system comprising a client and a server, wherein: The client includes: an acquisition module, a forwarding module, a search module, and a display module; An acquisition module, adapted to acquire a material search request sent by a request object; A forwarding module, adapted to forward the material search request to a server; The server is adapted to perform a video material search according to the material search request, obtain a video material search result, and return visual data corresponding to the video material search result to the client if the copyright information of the first video material corresponding to the video material search result is the first copyright information; A search module, adapted to search for local video material according to the visual data returned by the server to obtain a second video material; The display module is suitable for displaying the second video material to the request object so that the request object can create a video based on the second video material.

8. A computing device comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the video creation method as described in any one of claims 1-6.

9. A computer storage medium, wherein at least one executable instruction is stored in the storage medium, and the executable instruction enables a processor to perform operations corresponding to the video creation method as described in any one of claims 1-6.

10. A computer program product, comprising at least one executable instruction, wherein the executable instruction enables a processor to perform operations corresponding to the video creation method as described in any one of claims 1-6.

Images