Ship video multi-modal feature extraction method, ship video database and multi-modal query method

Abstract

The invention provides a ship video multi-modal feature extraction method, a ship video database and a multi-modal query method. The method comprises the following steps: S1, receiving a ship video and a first prompt word, wherein the first prompt word is a preset interested aspect of a ship; s2, according to the received ship video, based on the first prompt word, outputting an image frame and text data by a pre-trained large language model (LLM) module; s3, carrying out word segmentation on the output text data by a text word segmentation removal device to form description features, wherein the description features correspond to preset interested aspects of ships; s4, processing the output image frame and text data by a first linear layer to generate dense features; s5, classifying the output image frames by a pre-trained image frame classifier to obtain clear image frames; and S6, according to the obtained clear image frame, an image detection and identification module identifies fine features of the ship in the clear image frame, and the fine features indicate attributes of components of the ship.

Description

Technical Field

[0001] The present invention relates to the field of ship video retrieval, and in particular to a ship video multimodal feature extraction method, a ship video database, and a multimodal query method. Background Art

[0002] In recent years, the proportion of my country's marine economy in the national economy has continued to increase, and the content and scope of marine development and utilization have continued to expand. As a result, various illegal activities such as illegal sand mining, fishing, reclamation, and dumping have become increasingly rampant. According to statistics from 2002 to 2010, there were 17,073 cases of illegal use of sea areas and 5,006 cases of illegal marine environmental protection during the above period. At the same time, with the increase in international exchanges, some neighboring countries have used shipboard instruments and equipment of merchant ships and scientific research vessels to illegally obtain important strategic information such as seabed topography, hydrology and meteorology in the waters under my country's jurisdiction in the name of trade and scientific research, which has greatly harmed my country's marine rights and interests. The construction of a marine target surveillance and monitoring system is conducive to strengthening my country's capabilities in sea area and island management, marine ecological protection, marine rights and interests maintenance, and maritime security, and improving the level of comprehensive marine management.

[0003] In order to effectively monitor the country's coastline, exclusive economic zone and maritime targets, a business-based surveillance network based on air, space and ground integration has been gradually established. According to different detection methods, maritime target surveillance and monitoring systems can be roughly divided into passive and active types. The passive type mainly refers to obtaining the name, type, location, course, speed and other information of the target ship through the ship automatic identification system AIS, and then identifying the ship as cooperative or non-cooperative. The active type is mainly aimed at non-cooperative ships, using radar or optical images for detection and identification. However, due to the huge amount of captured data, the targeted recording effect for each ship is not good, and therefore it is impossible to effectively archive the captured data and conduct subsequent inquiries. Therefore, an effective archiving method for recording the situation of each ship is needed. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention provides a ship video multimodal feature extraction method, a ship video database, and a multimodal query method, which mainly solve the problem in the prior art of lacking an effective archiving method for recording the conditions of each ship.

[0005] The objective of the present invention is achieved through the following solutions:

[0006] In a first aspect, according to an embodiment of the present invention, a method for extracting multimodal features from a ship video is provided, comprising the following steps: S1, receiving a ship video and a first prompt word, wherein the first prompt word is a preset aspect of interest of the ship; S2, according to the received ship video and based on the first prompt word, a pre-trained large language model module outputs an image frame and text data; S3, a text de-segmenter performs word segmentation on the output text data to form a descriptive feature, wherein the descriptive feature corresponds to the preset aspect of interest of the ship; S4, a first linear layer processes the output image frame and text data to generate a dense feature; S5, a pre-trained image frame classifier classifies the output image frame to obtain a clear image frame; S6, according to the obtained clear image frame, an image detection and recognition module identifies fine features of the ship in the clear image frame, wherein the fine features indicate attributes of components of the ship.

[0007] According to an embodiment of the present invention, the preset interesting aspects of the ship are the ship perspective, the ship type, the hull color, the cabin color and the number of cabin layers.

[0008] According to an embodiment of the present invention, the hull color and the cabin color include one or more values.

[0009] According to an embodiment of the present invention, the method for extracting multimodal features from a ship video further includes: extracting key frames from the ship video to form a new ship video for subsequent processing.

[0010] According to an embodiment of the present invention, the method for extracting multimodal features from a ship video further includes: before the first prompt word is processed by the large language model module, segmenting the first prompt word by a text segmenter.

[0011] According to an embodiment of the present invention, the method for extracting multimodal features from a ship video further includes: encoding the ship video by a visual encoder and a second linear layer before the ship video is processed by the large language model module.

[0012] According to an embodiment of the present invention, the image frame classifier includes a third linear layer and a normalization layer.

[0013] According to an embodiment of the present invention, the normalization layer adopts a softmax function.

[0014] According to an embodiment of the present invention, the image frame classifier is trained based on a preset image frame data set and corresponding annotations.

[0015] According to an embodiment of the present invention, the method for extracting multimodal features from a ship video further includes: based on a preset video data set and a corresponding annotation description, the large language model module performs LoRA fine-tuning.

[0016] According to an embodiment of the present invention, only all linear layers are subjected to parameter updating.

[0017] According to an embodiment of the present invention, the image detection and recognition module includes a pre-trained bow and stern detection model, a ship marker detection model, a color recognition model, a ship text detection model, and a ship text recognition model, wherein the bow and stern detection model is used to identify the bow, hull and stern in the image frame, the ship marker detection model is used to identify the mast, flag, life buoy or tire, the color recognition model is used to further identify the color of the marker, and the ship text detection model and the ship text recognition model are used to detect ship plates or other texts and recognize the texts.

[0018] According to an embodiment of the present invention, the ship character recognition model is trained based on ship image data and corresponding annotations, wherein the ship image data includes images containing traditional Chinese characters, modified Chinese characters or modified characters.

[0019] According to an embodiment of the present invention, the method for extracting multimodal features from a ship video also includes: constructing a second prompt word with descriptive features and fine features, and obtaining video sparse features through a video sparse feature extraction module, wherein the video sparse feature extraction module includes a second text segmenter, a second large language model module and a fourth linear layer.

[0020] According to an embodiment of the present invention, the second large language model module and the fourth linear layer perform LoRA fine-tuning based on a preset feature data set, wherein the preset feature data set includes descriptive features and fine features.

[0021] In a second aspect, according to another embodiment of the present invention, there is provided a ship video database, including ship videos and a ship multimodal feature table, wherein the ship multimodal feature table is obtained based on a ship video multimodal feature extraction method as described in the first aspect.

[0022] In a third aspect, according to another embodiment of the present invention, there is provided a multimodal query method for querying the ship video database as described in the second aspect, the multimodal query method comprising the following steps: P1, detecting whether the multimodal input is text; P2, when the multimodal input is text, comparing the input text with the information in the multimodal feature table in the ship video database, and returning the query result; P3, when the multimodal input is a video or an image, using a ship video multimodal feature extraction method as described in the first aspect to obtain the multimodal features of the video or image, and comparing them with the information in the multimodal feature table, and returning the query result.

[0023] According to yet another embodiment of the present invention, when the input text or the acquired multimodal features include a ship license plate, a ship license plate comparison is performed.

[0024] According to yet another embodiment of the present invention, when the ship license plate comparison fails, sparse feature matching is performed.

[0025] According to another embodiment of the present invention, the query result is returned by using the first prompt word and outputting the return result by the large model output module based on the queried multimodal features.

[0026] In a fourth aspect, according to an embodiment of the present invention, a computer program product is provided, wherein the computer program product comprises a computer program code, and when the computer program code runs on an electronic device, the electronic device executes the method as described in the first aspect or the third aspect.

[0027] In a fifth aspect, according to an embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, and the computer program can be executed by a processor to implement the steps of the method described in the first aspect or the third aspect.

[0028] In the sixth aspect, according to an embodiment of the present invention, an electronic device is provided, comprising: one or more processors; and a memory, wherein the memory is used to store executable instructions; the one or more processors are configured to implement the steps of the method described in the first aspect or the third aspect by executing the executable instructions.

[0029] Compared with the prior art, the present invention has the following beneficial effects: by adopting a large model, video features can be better integrated with text, accurate analysis of complex scenes can be achieved, and efficient and intelligent solutions can be provided for maritime management, search and rescue, and security monitoring; in addition, the use of multimodal retrieval makes the application more extensive. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The embodiments of the present invention are further described below with reference to the accompanying drawings:

[0031] Figure 1 Schematic diagram of the process of a multimodal feature extraction method for a ship video according to an embodiment of the present invention;

[0032] Figure 2 Schematic diagram of a multimodal feature extraction process of a ship video according to an embodiment of the present invention;

[0033] Figure 3 is a schematic diagram of an image detection process according to an embodiment of the present invention;

[0034] Figure 4 Schematic diagram of a sparse feature extraction process according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] Before proceeding to the following detailed description, it may be advantageous to set forth the definitions of certain words and phrases used throughout this patent document. The terms "coupling", "connection" and their derivatives refer to any direct or indirect communication or connection between two or more elements, regardless of whether those elements are in physical contact with each other. The terms "transmission", "reception" and "communication" and their derivatives cover direct and indirect communication. The terms "include" and "comprising" and their derivatives refer to including but not limited to. The term "or" is inclusive, meaning and / or. The phrase "associated with..." and its derivatives refer to including, including within, interconnecting, containing, contained within, connecting or connecting with, coupling or coupling with, communicating with, cooperating, interweaving, parallel, close to, binding or binding with, having, having attributes, having a relationship or having a relationship with, etc. The term "controller" refers to any device, system or part thereof that controls at least one operation. Such a controller can be implemented with hardware, or a combination of hardware and software and / or firmware. The functions associated with any particular controller can be centralized or distributed, whether local or remote. The phrase "at least one of", when used with a list of items, means that different combinations of one or more of the listed items may be used, and only one of the items in the list may be required. For example, "at least one of A, B, C" includes any of the following combinations: A, B, C, A and B, A and C, B and C, A and B and C.

[0036] Definitions for other specific words and phrases are provided throughout this patent document. Those of ordinary skill in the art should understand that in many, if not most instances, such definitions apply to prior and future uses of such defined words and phrases.

[0037] In this patent document, the application combination of modules and the division level of sub-modules are only used for illustration, and the application combination of modules and the division level of sub-modules may have different forms without departing from the scope of the present disclosure.

[0038] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below through specific embodiments in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0039] As described in the background technology, the prior art lacks an effective archiving method for recording the status of each ship. Figure 1 and Figure 2As shown, according to an embodiment of the present invention, a method for extracting multimodal features from a ship video is provided, comprising the following steps: S1, receiving a ship video and a first prompt word, wherein the first prompt word is a preset aspect of interest of the ship; S2, according to the received ship video, based on the first prompt word, a pre-trained large language model (LLM) module outputs an image frame and text data; S3, a text de-segmenter performs word segmentation on the output text data to form a description feature, wherein the description feature corresponds to the preset aspect of interest of the ship; S4, a first linear layer processes the output image frame and text data to generate a dense feature; S5, a pre-trained image frame classifier classifies the output image frame to obtain a clear image frame; S6, according to the obtained clear image frame, an image detection and recognition module identifies fine features of the ship in the clear image frame, wherein the fine features indicate attributes of the components of the ship. By adopting a large model, video features can be better integrated with text, achieving accurate analysis of complex scenes, and providing efficient and intelligent solutions for maritime management, search and rescue, and security monitoring; in addition, the use of multimodal retrieval makes the application more extensive.

[0040] According to an embodiment of the present invention, the preset aspects of interest of a ship are ship perspective, ship type, hull color, cabin color and number of cabin layers. The large model generates descriptive features from video sequence features. The first prompt word (prompt word 1) can be: "Briefly describe the ship features from 5 aspects, the reply format is as follows: \nShip perspective: {Select one: left, right, front, back} \nShip type: {Select one: large cargo ship, small cargo ship, large passenger ship, small passenger ship, large fishing boat, small fishing boat, special ship, other} \nHull color: {red, yellow, blue, orange, green, purple, black, white, gray, pink, brown, cyan} \nCabin color: {red, yellow, blue, orange, green, purple, black, white, gray, pink, brown, cyan} \nNumber of cabin layers: {n}". According to the format of prompt word 1, the LLM model outputs the descriptive features (shipPerspective, shipType, shipColor, shipCabin) in the structured multimodal feature table.

[0041] For attributes that have combinations, in order to better represent the attributes, according to an embodiment of the present invention, the hull color and the cabin color include one or more values.

[0042] In addition, in order to reduce redundant visual information, according to an embodiment of the present invention, the method for extracting multimodal features from a ship video further includes: extracting key frames from the ship video to form a new ship video for subsequent processing. The ship video is a video stream obtained by a surveillance camera from a fixed angle.

[0043] In natural language processing (NLP), in order to provide a better basis for subsequent analysis, understanding and text generation of input text, according to an embodiment of the present invention, the method for extracting multimodal features from ship videos further includes: before the first prompt word is processed by the large language model module, the first prompt word is segmented by a text segmenter.

[0044] Similarly, in order to better capture the features of the video so that the large language model module can perform targeted processing, according to an embodiment of the present invention, the method for extracting multimodal features from a ship video also includes: encoding the ship video by a visual encoder and a second linear layer before the ship video is processed by the large language model module.

[0045] In addition, in order to classify the quality of the key frame images in the video (whether the key objects and parts on the ship are clear) in an image classification manner. According to an embodiment of the present invention, the image frame classifier includes a third linear layer and a normalization layer. According to an embodiment of the present invention, the normalization layer adopts a softmax function. The output of the LLM module is mapped through a linear layer and a Softmax layer to obtain the confidence of whether each key frame image is clear, and the image with the highest confidence is selected to extract image features. In addition, in order to make the classification effect of the image frame classifier better, according to an embodiment of the present invention, the image frame classifier is trained based on a preset image frame data set and corresponding annotations.

[0046] In addition, in order to make the LLM model more suitable for the ship video feature extraction in the present invention, according to an embodiment of the present invention, the ship video multimodal feature extraction method further includes: based on a preset video data set and a corresponding annotation description, the large language model module performs LoRA fine-tuning. According to an embodiment of the present invention, only all linear layers perform parameter updates. In detail, the visual encoder, text segmenter, and text de-segmenter are frozen, and all parameters of the three linear layers participate in the training.

[0047] After obtaining a clear key frame, such as Figure 3As shown, according to an embodiment of the present invention, the image detection and recognition module includes a pre-trained bow and stern detection model, a ship marker detection model, a color recognition model, a ship text detection model, and a ship text recognition model, wherein the bow and stern detection model is used to identify the bow, hull, and stern in the image frame, the ship marker detection model is used to identify the mast, flag, life buoy, or tire, the color recognition model is used to further identify the color of the marker, and the ship text detection model and the ship text recognition model are used to detect ship plates or other texts and recognize the texts. The key frame is input into the image detection and recognition module to obtain fine features (shipPlate, shipMast, shipFlag, shipText, shipLifebuoy). As shown in Table 1, the aforementioned descriptive features and fine features constitute a structured multimodal feature table.

[0048] Table 1. Ship structured multimodal features

[0049]

[0050] In addition, the color recognition model is an image color classification model based on a convolutional neural network (CNN), which is used to identify the main colors of masts, flags, life buoys or tires, and the color categories include: red, yellow, blue, orange, green, purple, black, white, gray, pink, brown, cyan. Similarly, the color can be one color or a combination of multiple colors.

[0051] In addition, in order to improve the detection and recognition effects, according to an embodiment of the present invention, the ship character recognition model is trained based on ship image data and corresponding annotations, wherein the ship image data includes images containing traditional Chinese characters, variant Chinese characters or variant characters.

[0052] According to an embodiment of the present invention, Figure 4 As shown, the multimodal feature extraction method for ship video also includes: constructing a second prompt word with descriptive features and fine features, and obtaining video sparse features through a video sparse feature extraction module, wherein the video sparse feature extraction module includes a second text segmenter, a second large language model module and a fourth linear layer. The second prompt word (prompt word 2) can be implemented as: "Generate an accurate ship description based on the following ship description information: \n{shipPerspective}, {shipPlate}, {shipType}, {shipColor}, {shipCabin}, {shipMast}, {shipFlag}, {shipText}, {shipLifebuoy}". Video sparse features can very accurately represent the visual features of the ship and are key features for query retrieval.

[0053] In addition, in order to better perform the function of generating the video sparse features, according to an embodiment of the present invention, the second large language model module and the fourth linear layer perform LoRA fine-tuning based on a preset feature data set, wherein the preset feature data set includes descriptive features and fine features.

[0054] According to another embodiment of the present invention, there is provided a ship video database, including ship videos and a ship multimodal feature table, wherein the ship multimodal feature table is obtained based on a ship video multimodal feature extraction method as described above.

[0055] In addition, if the same ship license plate and ship perspective exist in the database, the confidence of the refined features is compared one by one, and the feature information with low confidence of the ship at this perspective in the database is updated.

[0056] According to another embodiment of the present invention, a multimodal query method is provided for querying the ship video database as described above, and the multimodal query method comprises the following steps: P1, detecting whether the multimodal input is text; P2, when the multimodal input is text, comparing the input text with the information in the multimodal feature table in the ship video database, and returning the query result; P3, when the multimodal input is a video or an image, using the ship video multimodal feature extraction method as described above to obtain the multimodal features of the video or image, and comparing them with the information in the multimodal feature table, and returning the query result.

[0057] Multimodal inputs include video, image, and text. Video and image queries: Image queries can be viewed as videos with keyframe 1. Video and image inputs can be converted into structured multimodal feature tables and then queried in the database.

[0058] In order to improve the efficiency of the query, starting from the most specific feature comparison, according to another embodiment of the present invention, when the input text or the acquired multimodal features include a ship license plate, a ship license plate comparison is performed.

[0059] According to yet another embodiment of the present invention, when the ship license plate comparison fails, sparse feature matching is performed.

[0060] In order to better display the output, according to another embodiment of the present invention, the query result is returned as follows: using the first prompt word, based on the queried multimodal features, the large model output module outputs the return result.

[0061] According to an embodiment of the present invention, a computer program product is provided, which includes a computer program code. When the computer program code is executed on an electronic device, the electronic device executes a multimodal feature extraction method for a ship video or a multimodal query method as described above.

[0062] According to an embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. The computer program can be executed by a processor to implement the steps of a multimodal feature extraction method for ship videos or a multimodal query method as described above.

[0063] According to an embodiment of the present invention, there is provided an electronic device, comprising: one or more processors; and a memory, wherein the memory is used to store executable instructions; the one or more processors are configured to implement the steps of a multimodal feature extraction method for ship videos or a multimodal query method as described above by executing the executable instructions.

[0064] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0065] Computer readable storage medium can be a tangible device that holds and stores instructions used by an instruction execution device. Computer readable storage medium can include, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a protruding structure in a groove on which instructions are stored, and any suitable combination thereof.

[0066] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for extracting multimodal features from ship videos, characterized in that: The following steps are involved: S1, receiving a ship video and a first prompt word, where the first prompt word is a preset aspect of interest of the ship; S2. Based on the received ship video and the first prompt word, the pre-trained large language model module outputs image frames and text data; S3, using a text de-segmenter to segment the output text data to form descriptive features, where the descriptive features correspond to the preset aspects of interest of the vessel; S4, processing the output image frame and text data by a first linear layer to generate dense features; S5, classifying the output image frames by a pre-trained image frame classifier to obtain clear image frames; S6. Based on the obtained clear image frame, the image detection and recognition module recognizes fine features of the ship in the clear image frame, wherein the fine features indicate properties of components of the ship.

2. A method for extracting multimodal features from a ship video according to claim 1, characterized in that: The preset interesting aspects of a ship are ship perspective, ship type, hull color, cabin color, and number of cabin levels.

3. A method for extracting multimodal features from ship videos according to claim 2, characterized in that: The hull color and the cabin color include one or more values.

4. A method for extracting multimodal features from a ship video according to claim 1, characterized in that: Also includes: Key frames are extracted from the ship video to form a new ship video for subsequent processing.

5. The method for extracting multimodal features from a ship video according to claim 1, characterized in that: Also includes: Before the first prompt word is processed by the large language model module, the first prompt word is segmented by a text segmenter.

6. A method for extracting multimodal features from a ship video according to claim 1, characterized in that: Also includes: The ship video is encoded by a visual encoder and a second linear layer before being processed by the large language model module.

7. A method for extracting multimodal features from ship videos according to claim 6, characterized in that: The image frame classifier includes a third linear layer and a normalization layer.

8. A method for extracting multimodal features from ship videos according to claim 7, characterized in that: The normalization layer adopts the softmax function.

9. A method for extracting multimodal features from a ship video according to claim 7, characterized in that: The image frame classifier is trained based on a preset image frame data set and corresponding annotations.

10. The method for extracting multimodal features from a ship video according to claim 7, characterized in that: Also includes: Based on the preset video data set and the corresponding annotation description, the large language model module performs LoRA fine-tuning.

11. A method for extracting multimodal features from a ship video according to claim 10, characterized in that: Only all linear layers undergo parameter updates.

12. The method for extracting multimodal features from a ship video according to claim 1, characterized in that: The image detection and recognition module includes a pre-trained ship bow and stern detection model, a ship marker detection model, a color recognition model, a ship text detection model, and a ship text recognition model. Among them, the bow and stern detection model is used to identify the bow, hull and stern in the image frame, the ship marker detection model is used to identify the mast, flag, life buoy or tire, the color recognition model is used to further identify the color of the marker, and the ship text detection model and ship text recognition model are used to detect ship plates or other texts and recognize the text.

13. The method for extracting multimodal features from a ship video according to claim 1, characterized in that: The ship character recognition model is trained based on ship image data and corresponding annotations, wherein the ship image data includes images containing traditional Chinese characters, modified Chinese characters or modified characters.

14. A method for extracting multimodal features from a ship video according to claim 1, characterized in that: Also includes: The second prompt word is constructed by using the descriptive features and the fine features, and the video sparse features are obtained by a video sparse feature extraction module, wherein the video sparse feature extraction module includes a second text segmenter, a second large language model module and a fourth linear layer.

15. A method for extracting multimodal features from ship videos according to claim 14, characterized in that: The second largest language model module and the fourth linear layer perform LoRA fine-tuning based on a preset feature data set, wherein the preset feature data set includes descriptive features and fine features.

16. A vessel video database, characterized in that: It comprises a ship video and a ship multimodal feature table, wherein the ship multimodal feature table is obtained based on a ship video multimodal feature extraction method according to any one of claims 1-15.

17. A multimodal query method for querying the vessel video database according to claim 16, characterized in that: The multimodal query method comprises the following steps: P1. Detect whether the multimodal input is text; P2. When the multimodal input is text, compare the input text with the information in the multimodal feature table in the vessel video database, and return the query result; P3. When the multimodal input is a video or an image, a ship video multimodal feature extraction method as described in any one of claims 1 to 15 is used to obtain the multimodal features of the video or image, and the multimodal features are compared with the information in the multimodal feature table, and the query results are returned.

18. A multimodal query method for a vessel video database according to claim 17, characterized in that: When the input text or the acquired multimodal features include a ship license plate, a ship license plate comparison is performed.

19. A multimodal query method for a vessel video database according to claim 18, characterized in that: When the ship license plate comparison fails, sparse feature matching is performed.

20. The multimodal query method for a vessel video database according to claim 17, characterized in that: The returned query result is: using the first prompt word, based on the queried multimodal features, the large model output module outputs the returned result.

21. A computer program product, characterized in that The computer program product comprises a computer program code, and when the computer program code is run on an electronic device, the electronic device executes the method as claimed in any one of claims 1 to 15 and 17 to 20.

22. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of the method in any one of claims 1-15, 17-20.

23. An electronic device, characterized in that: include: one or more processors; as well as A memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method of any one of claims 1-15, 17-20 by executing the executable instructions.

Images