Ship water gauge reading identification method based on video understanding large model

Abstract

The invention discloses a ship water gauge reading identification method based on a video understanding large model, and relates to the technical field of large models and video processing, and the method comprises the steps: obtaining a water gauge region video of a ship to be read and a question text given by a user, determining a water gauge reading result corresponding to the ship to be read through the video understanding large model; the training process of the video understanding large model specifically comprises the following steps: acquiring a water gauge video of each region of an experimental ship, performing equal-interval frame extraction based on the videos to obtain an RGB image set, and completing visual coding; converting an experiment problem text aiming at the experiment ship to obtain a text feature vector; and carrying out alignment fusion on the text feature vector and the visual coding feature to obtain an output result. The large model is understood through the video, and the problem of accumulative errors caused by complex programs due to multi-module linkage in a traditional machine vision method can be avoided.

Description

Background Art

[0002] Ship draft gauge weighing is a method of measuring the cargo on board. This method is widely used in the weight measurement of large quantities of cheap bulk cargo transported by ships. The accuracy of ship draft gauge reading recognition directly affects the accuracy of ship cargo calculation, which in turn affects the settlement of commodity trade and customs clearance tax calculation, etc. Therefore, it is of great significance to ensure the accuracy of ship draft gauge line detection and draft gauge reading recognition. At present, the measurement of ship draft gauge line mainly includes the following methods:

[0003] Manual observation method. After the ship docks, experienced water gauge observers will conduct manual observation, record and check, compare and average the observation results to get the final result. This method is greatly affected by subjective and objective factors. For example, it is difficult to observe the water gauge reading when the water surface fluctuates greatly. Due to the limitations of weather and observation angle, observers cannot obtain accurate values. Secondly, the efficiency of manual observation of water gauge readings is low and cannot meet the efficient passage of ports.

[0004] Ship water level detection method based on deep learning algorithm. With the rapid development of deep learning algorithms, more and more models based on convolutional neural networks are used for the detection and recognition of ship water level readings. This type of method first corrects the original input image through affine transformation; secondly, the ship water level area and the ship-water boundary line position are extracted using the instance segmentation algorithm; then the water level area image is sent to the OCR character recognition algorithm to read the water level value; finally, the integer and decimal values ​​of the water level are determined through complex post-processing logic and the final water level reading is obtained by combining the ship-water boundary line position estimation. This type of method has high requirements on the accuracy of image preprocessing and instance segmentation algorithms. In practical applications, water level readings are usually distorted due to tilted shooting angles, curved hull surfaces, etc. In this case, the accuracy of this type of method is not high; secondly, the complex post-processing logic can usually only meet some specific scenarios. When the ship type or monitoring scenario changes, the post-processing algorithm needs to be redesigned. Therefore, the method of determining integer and decimal values ​​through post-processing algorithms has low generalization performance and cannot be deployed on a large scale. Summary of the invention

[0005] The technical problem to be solved by the present invention is to address the deficiencies of the prior art and specifically provide a method for identifying ship water gauge readings based on a large video understanding model, which is as follows:

[0006] 1) In the first aspect, the present invention provides a method for identifying ship water gauge readings based on a large video understanding model, and the specific technical solution is as follows:

[0007] Obtain a video of a water gauge area of ​​a ship to be read and a question text given by a user, and process the video of the water gauge area and the question text through a large video understanding model to obtain a water gauge reading result corresponding to the ship to be read;

[0008] The training process of the video understanding large model is specifically as follows:

[0009] The water level video of each area of ​​the experimental ship is obtained respectively, and each water level video is subjected to frame extraction at equal intervals to obtain the RGB image set corresponding to each water level video. Each RGB image set is subjected to visual coding processing to obtain visual coding features.

[0010] Transform the text of the experimental question for the experimental ship to obtain a text feature vector;

[0011] After aligning and fusing the text feature vector with the visual encoding feature, the output result is obtained.

[0012] The beneficial effects of the ship water gauge reading recognition method based on the video understanding large model provided by the present invention are as follows:

[0013] The video understanding big model can realize the recognition of water gauge readings in any water gauge area video collected on the ship. In addition, the video understanding big model can also avoid the problem of cumulative errors caused by complex programs caused by the linkage of multiple modules in traditional machine vision methods.

[0014] Based on the above solution, the present invention can also be improved as follows.

[0015] Furthermore, the specific process of performing visual encoding processing on each RGB image set is as follows:

[0016] A convolution operation is performed on each RGB image in each RGB image set to obtain a feature matrix corresponding to each RGB image, and the feature matrices corresponding to all RGB images are spliced ​​to obtain a splicing result corresponding to any RGB image set, and the splicing result is used as the visual coding feature.

[0017] Furthermore, the process of equally spaced frame extraction for each training water gauge area video is as follows:

[0018] OpenCV is used to extract frames at equal intervals for each training water gauge area video.

[0019] Furthermore, the text feature vector is aligned and fused with the visual coding feature and then input into the large model to obtain the output result as follows:

[0020] The visual coding features are projected into a data feature space having the same data dimension as the text feature vector. In the data feature space, the text feature vector and the visual coding features are aligned and fused, and then input into a large model to obtain an output result.

[0021] 2) In the second aspect, the present invention also provides a ship water gauge reading recognition system based on a video understanding large model, and the specific technical solution is as follows:

[0022] The recognition module is used to: obtain a video of the water gauge area of ​​the ship to be read and a question text given by the user, and process the video of the water gauge area and the question text through the video understanding large model to obtain the water gauge reading result corresponding to the ship to be read;

[0023] The training process of the video understanding large model is specifically as follows:

[0024] The water level video of each area of ​​the experimental ship is obtained respectively, and each water level video is subjected to frame extraction at equal intervals to obtain the RGB image set corresponding to each water level video. Each RGB image set is subjected to visual coding processing to obtain visual coding features.

[0025] Transform the text of the experimental question for the experimental ship to obtain a text feature vector;

[0026] After aligning and fusing the text feature vector with the visual encoding feature, the output result is obtained.

[0027] Based on the above solution, the present invention can also be improved as follows.

[0028] Furthermore, the specific process of performing visual encoding processing on each RGB image set is as follows:

[0029] A convolution operation is performed on each RGB image in each RGB image set to obtain a feature matrix corresponding to each RGB image, and the feature matrices corresponding to all RGB images are spliced ​​to obtain a splicing result corresponding to any RGB image set, and the splicing result is used as the visual coding feature.

[0030] Furthermore, the process of equally spaced frame extraction for each training water gauge area video is as follows:

[0031] OpenCV is used to extract frames at equal intervals for each training water gauge area video.

[0032] Furthermore, the text feature vector is aligned and fused with the visual coding feature and then input into the large model to obtain the output result as follows:

[0033] The visual coding features are projected into a data feature space having the same data dimension as the text feature vector. In the data feature space, the text feature vector and the visual coding features are aligned and fused, and then input into a large model to obtain an output result.

[0034] 3) In a third aspect, the present invention further provides an electronic device, comprising a processor, wherein the processor is coupled to a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor so that the electronic device implements any of the above methods.

[0035] 4) In a fourth aspect, the present invention further provides a computer-readable storage medium, wherein at least one computer program is stored in the computer-readable storage medium, and the at least one computer program is loaded and executed by a processor so that a computer implements any of the above methods.

[0036] It should be noted that the beneficial effects achieved by the technical solutions of the second to fourth aspects of the present invention and the corresponding possible implementation methods can be found in the above-mentioned technical effects of the first aspect and its corresponding possible implementation methods, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0038] Figure 1 A schematic diagram of a flow chart of a method for identifying ship water gauge readings based on a large video understanding model according to an embodiment of the present invention;

[0039] Figure 2 A schematic diagram of a model architecture of a method for identifying ship water gauge readings based on a large video understanding model according to an embodiment of the present invention;

[0040] Figure 3 The figure is a structural framework diagram of an electronic device. DETAILED DESCRIPTION

[0041] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0042] like Figure 1 as well as Figure 2 As shown, a method for identifying ship water gauge readings based on a video understanding large model according to an embodiment of the present invention comprises the following steps:

[0043] S1, obtaining a video of a water gauge area of ​​a ship to be read and a question text given by a user, and processing the video of the water gauge area and the question text through a video understanding large model to obtain a water gauge reading result corresponding to the ship to be read;

[0044] The training process of the video understanding large model is specifically as follows:

[0045] The water level video of each area of ​​the experimental ship is obtained respectively, and each water level video is subjected to frame extraction at equal intervals to obtain the RGB image set corresponding to each water level video. Each RGB image set is subjected to visual coding processing to obtain visual coding features.

[0046] Transform the text of the experimental question for the experimental ship to obtain a text feature vector;

[0047] After aligning and fusing the text feature vector with the visual encoding feature, the output result is obtained.

[0048] The beneficial effects of the ship water gauge reading recognition method based on the video understanding large model provided by the present invention are as follows:

[0049] The video understanding big model can realize the recognition of water level readings in any water level area video collected on the ship. In addition, the video understanding big model can also avoid the problem of excessive cost consumption caused by complex procedures caused by the linkage of multiple modules in traditional machine vision methods.

[0050] The water gauge area video of the ship to be read may be a water gauge area video corresponding to at least one of the bow, midship and stern. It should be further explained that it may also be a combination of multiple videos, such as a water gauge area video of the bow and a water gauge area video of the midship.

[0051] The question text given by the user refers to the question entered by the user according to the format requirements.

[0052] Provide format examples and question examples in a prominent position above or below the area where users enter question text.

[0053] The obvious position refers to an area of ​​text displayed in a red or highlighted manner, located at a preset distance above or below the input area in the current display area of ​​the display terminal.

[0054] Experimental ships refer to ships used to provide training samples for model training.

[0055] The acquired training water gauge area videos of the bow, midship and stern of the experimental ship were preprocessed, and the training water gauge area videos at each position were processed by equally spaced frames through OpenCV to obtain an RGB image set consisting of 8 fixed RGB images for each video. Each RGB image set was visually encoded using VIT-H / 14.

[0056] The preprocessing process specifically includes:

[0057] Determine whether there is a bright area with a length exceeding the preset length in any RGB image in any RGB image set. If so, further determine whether the bright area blocks the water gauge. If the judgment result is not blocked, adjust the brightness of the RGB image to make the model more conducive to extracting the water gauge reading.

[0058] If the judgment result is occlusion, the positions of the RGB image and the two RGB images adjacent to the RGB image in the training water scale area video are determined. Within the range formed by the above three positions, the image that is closest to the RGB image and does not occlude the water scale is determined frame by frame, and the image is determined as a new RGB image. The new RGB image is used as the RGB image with the judgment result of occlusion for subsequent processing.

[0059] It should be further explained that in practical applications, if a high-resolution image is directly encoded, a feature vector with a very large dimension will be generated. In particular, in video understanding tasks, it is also necessary to perform feature fusion on multiple frames of images and text token sequences. At this time, the input length will exceed the context length of the large model, and the training time and computational overhead of the model will increase significantly. To this end, the present invention uses a convolutional neural network to reduce the dimension of a single image to obtain a feature matrix. Subsequently, the feature matrices corresponding to all video frames are spliced ​​on the first dimension to achieve inter-frame feature fusion. The visual encoding operation achieved in this way not only includes the feature information of the entire video, but also reduces the data dimension of the large model input, reduces the computational overhead of the model training and fine-tuning stages, and can also improve computational efficiency in the actual reasoning stage. Among them, the first dimension refers to the number of rows of the feature matrix. The dimensions of the feature matrices of different video frames are the same after convolution operations, and can be spliced ​​on the number of rows to achieve inter-frame feature fusion.

[0060] A video understanding large model usually refers to a large model that can give the understanding of the video in the form of text and answer the user's questions based on the video and questions given by the user. The video understanding dataset includes the original ship draft gauge video and the corresponding text question and answer tags. The text question and answer tag consists of two parts, namely the question text and the answer text. The question text is also called a prompt, for example, "Describe this video and answer how many meters the ship draft gauge reading is", and the answer text describes the video content in concise and accurate language and answers the question text. In the present invention, multiple semantically similar question texts are set for the ship draft gauge reading task, and experienced ship draft gauge observers give accurate reading results as answer texts to form a ship draft gauge reading video understanding dataset.

[0061] In addition, it should be further explained that the input of the video understanding large model can be multiple videos of one position (any one of the bow, midship and stern) or videos of multiple positions. That is, in actual applications, users can upload water gauge videos of different areas of the same ship multiple times to let the large model give the water gauge reading of each video, and in the last round of dialogue, let the large model give the final water gauge reading based on the previous answers. This ability is the result of the large language model training based on massive data sets. The ship water gauge reading task is fine-tuned and trained on the basis of the original large model, so that it has higher accuracy in the water gauge reading recognition task.

[0062] Furthermore, the specific process of performing visual encoding processing on each RGB image set is as follows:

[0063] A convolution operation is performed on each RGB image in each RGB image set to obtain a feature matrix corresponding to each RGB image, and the feature matrices corresponding to all RGB images are spliced ​​to obtain a splicing result corresponding to any RGB image set, and the splicing result is used as the visual coding feature.

[0064] Furthermore, the process of equally spaced frame extraction for each training water gauge area video is as follows:

[0065] OpenCV is used to extract frames at equal intervals for each training water gauge area video.

[0066] Furthermore, the text feature vector is aligned and fused with the visual coding feature and then input into the large model to obtain the output result as follows:

[0067] The visual coding features are projected into a data feature space having the same data dimension as the text feature vector. In the data feature space, the text feature vector and the visual coding features are aligned and fused, and then input into a large model to obtain an output result.

[0068] The projection process is as follows: the projection layer is usually a linear layer (LinearLayer), and the projection operation is implemented through matrix multiplication to map the matrix dimension corresponding to the visual encoding to the same dimension as the question text sequence. For example, assuming that the matrix dimension corresponding to a visual encoding is [8192, 1280], and the dimension of the question text sequence is [1, 1024], then the visual encoding matrix can be multiplied with a projection matrix of dimension [1280, 1024] to become [8192, 1024]. At this time, the second dimension (1024) of the visual encoding matrix is ​​the same as the second dimension of the text sequence. Then, a concatenation operation (8192+1) can be performed on the first dimension to obtain a fused matrix (dimension [8193, 1024]) as the input of the large model. If the visual encoding and the question text sequence are different in the first and second dimensions, feature fusion cannot be achieved.

[0069] In Example 1, drones are used to collect videos of the bow, midship and stern water gauge areas. In order to ensure that the training and fine-tuning of the subsequent video understanding large model achieves the ideal effect, the collected data needs to cover scenes with different lighting, weather and ship types, and the ship's water gauge scale and ship-water boundary line in the picture are clearly visible. According to practical experience, the ship water gauge video can usually be collected for 30-60 seconds to achieve accurate water gauge recognition.

[0070] A ship draft gauge video understanding dataset is created. For each draft gauge video, a question text is set, such as "What is the ship draft gauge reading in this video?" At the same time, experienced draft gauge observers give the accurate draft gauge reading of the video (with an error accuracy of less than 0.01 meter) as label data, i.e., the answer text.

[0071] The water level video is sampled at equal intervals. The present invention samples N=8 frames of image data for any length of video. The sampling interval interval is determined by the video length and frame rate. For example, if the video frame rate FPS=25 and the video length is 30 seconds, the sampling interval t=(T×FPS) / N is 93.75 frames. The original image resolution is 448×448. After convolution operation, the feature maps corresponding to the 8 sampled video frames are spliced. The specific formula is as follows:

[0072]

[0073] f v =Concat(f1...f n )

[0074] in, represents the nth image data, n≤8, H and W represent the height and width of the original image, both are 448, and C is the number of image channels. Conv represents the convolution operation, the convolution kernel size is 14×14, the step size is 14, the number of convolution kernels is 1280, and f n represents the image features after convolution operation, Concat represents the concatenation operation, and f v It is the video encoding vector of 8 images concatenated in the first dimension.

[0075] Bert-tokenizer is used for sequence encoding of question text. The sequence encoder includes word embedding layer and position embedding layer. The word embedding layer converts each token into a vector of fixed size based on the index ID of each token in the vocabulary. Since Bert is based on the Transformer architecture, it is necessary to introduce a position embedding layer to provide each token with its position information in the sentence to help the large language model understand the order of words, which plays an important role in the watermark video question answering task.

[0076] The video encoding vector is mapped to the same dimensional space as the text token vector through a Linear projection layer, and then the text token vector and the video encoding vector are input into the large language model together to predict the specific reading of the current water gauge video.

[0077] Through multiple rounds of training iterations, the parameters of the video understanding large model are updated to achieve the prediction of the ship water level reading task.

[0078] In the above embodiments, although the steps are numbered S1, S2, etc., these are only specific embodiments given by the present invention. Those skilled in the art may adjust the execution order of S1, S2, etc. according to actual conditions, which is also within the protection scope of the present invention. It can be understood that in some embodiments, some or all of the above embodiments may be included.

[0079] The present invention also provides a ship water gauge reading recognition system based on a video understanding large model, and the specific technical solution is as follows:

[0080] The recognition module is used to: obtain a video of the water gauge area of ​​the ship to be read and a question text given by the user, and process the video of the water gauge area and the question text through the video understanding large model to obtain the water gauge reading result corresponding to the ship to be read;

[0081] The training process of the video understanding large model is specifically as follows:

[0082] The water level video of each area of ​​the experimental ship is obtained respectively, and each water level video is subjected to frame extraction at equal intervals to obtain the RGB image set corresponding to each water level video. Each RGB image set is subjected to visual coding processing to obtain visual coding features.

[0083] Transform the text of the experimental question for the experimental ship to obtain a text feature vector;

[0084] After aligning and fusing the text feature vector with the visual encoding feature, the output result is obtained.

[0085] It should be noted that the beneficial effects of the ship draft reading recognition system based on the video understanding large model provided in the above embodiment are the same as the beneficial effects of the ship draft reading recognition method based on the video understanding large model, which will not be repeated here. In addition, when the system provided in the above embodiment realizes its functions, it only uses the division of the above functional modules as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the system can be divided into different functional modules according to actual conditions to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0086] like Figure 3 As shown, an electronic device 300 according to an embodiment of the present invention includes a processor 320, the processor 320 is coupled to a memory 310, and the memory 310 stores at least one computer program 330, and the at least one computer program 330 is loaded and executed by the processor 320, so that the electronic device 300 implements any of the above methods, specifically:

[0087] The electronic device 300 may have relatively large differences due to different configurations or performances, and may include one or more processors 320 (Central Processing Units, CPU) and one or more memories 310, wherein the one or more memories 310 store at least one computer program 330, and the at least one computer program 330 is loaded and executed by the one or more processors 320, so that the electronic device 300 implements a ship water gauge reading recognition method based on a large video understanding model provided in the above embodiment. Of course, the electronic device 300 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output, and the electronic device 300 may also include other components for realizing device functions, which will not be repeated here.

[0088] A computer-readable storage medium according to an embodiment of the present invention stores at least one computer program, and the at least one computer program is loaded and executed by a processor so that a computer implements any one of the above methods.

[0089] Optionally, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0090] In an exemplary embodiment, a computer program product or a computer program is also provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs any of the above methods.

[0091] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects and represent the definition of a specific order or sequence. The order of use of similar objects can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than the order shown or described.

[0092] Those skilled in the art know that the present invention can be implemented as a system, method or computer program product. Therefore, the present disclosure can be specifically implemented in the following forms, namely: it can be complete hardware, it can be complete software (including firmware, resident software, microcode, etc.), or it can be a combination of hardware and software, which is generally referred to as "circuit", "module" or "system" herein. In addition, in some embodiments, the present invention can also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable medium contains computer-readable program code.

[0093] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device, or device.

[0094] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A method for identifying ship water gauge readings based on a large video understanding model, characterized in that: include: Obtain a video of a water gauge area of ​​a ship to be read and a question text given by a user, and process the video of the water gauge area and the question text through a large video understanding model to obtain a water gauge reading result corresponding to the ship to be read; The training process of the video understanding large model is specifically as follows: The water level video of each area of ​​the experimental ship is obtained respectively, and each water level video is subjected to frame extraction at equal intervals to obtain the RGB image set corresponding to each water level video. Each RGB image set is subjected to visual coding processing to obtain visual coding features. Transform the text of the experimental question for the experimental ship to obtain a text feature vector; After aligning and fusing the text feature vector with the visual encoding feature, the output result is obtained.

2. According to claim 1, a method for identifying ship water gauge readings based on a large video understanding model is characterized in that: The specific process of performing visual encoding processing on each RGB image set is as follows: A convolution operation is performed on each RGB image in each RGB image set to obtain a feature matrix corresponding to each RGB image, and the feature matrices corresponding to all RGB images are spliced ​​to obtain a splicing result corresponding to any RGB image set, and the splicing result is used as the visual coding feature.

3. According to claim 1, a method for identifying ship water gauge readings based on a large video understanding model is characterized in that: The specific process of equally spaced frame extraction for each training water gauge area video is as follows: OpenCV is used to extract frames at equal intervals for each training water gauge area video.

4. According to the method of claim 1, the ship water gauge reading recognition method based on the video understanding large model is characterized in that: The text feature vector is aligned and fused with the visual coding feature and then input into the large model to obtain the output result as follows: The visual coding features are projected into a data feature space having the same data dimension as the text feature vector. In the data feature space, the text feature vector and the visual coding features are aligned and fused, and then input into a large model to obtain an output result.

5. A ship water gauge reading recognition system based on a large video understanding model, characterized in that: include: The recognition module is used to: obtain a video of the water gauge area of ​​the ship to be read and a question text given by the user, and process the video of the water gauge area and the question text through the video understanding large model to obtain the water gauge reading result corresponding to the ship to be read; The training process of the video understanding large model is specifically as follows: The water level video of each area of ​​the experimental ship is obtained respectively, and each water level video is subjected to frame extraction at equal intervals to obtain the RGB image set corresponding to each water level video. Each RGB image set is subjected to visual coding processing to obtain visual coding features. Transform the text of the experimental question for the experimental ship to obtain a text feature vector; After aligning and fusing the text feature vector with the visual encoding feature, the output result is obtained.

6. A ship water gauge reading recognition system based on a video understanding large model according to claim 5, characterized in that: The specific process of performing visual encoding processing on each RGB image set is as follows: A convolution operation is performed on each RGB image in each RGB image set to obtain a feature matrix corresponding to each RGB image, and the feature matrices corresponding to all RGB images are spliced ​​to obtain a splicing result corresponding to any RGB image set, and the splicing result is used as the visual coding feature.

7. The ship water gauge reading recognition system based on video understanding large model according to claim 5 is characterized in that: The specific process of equally spaced frame extraction for each training water gauge area video is as follows: OpenCV is used to extract frames at equal intervals for each training water gauge area video.

8. The ship water gauge reading recognition system based on video understanding large model according to claim 5 is characterized in that: The text feature vector is aligned and fused with the visual coding feature and then input into the large model to obtain the output result as follows: The visual coding features are projected into a data feature space having the same data dimension as the text feature vector. In the data feature space, the text feature vector and the visual coding features are aligned and fused, and then input into a large model to obtain an output result.

9. An electronic device, characterized in that: The electronic device comprises a processor, the processor is coupled to a memory, at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor so that the electronic device implements the method according to any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by a processor so that a computer implements the method according to any one of claims 1 to 4.

Images