Ship image data set construction method and device, equipment and storage medium

Abstract

The invention belongs to the technical field of computers, and particularly relates to a ship image data set construction method and device, equipment and a storage medium. According to the method, a keyword set of a ship is obtained, the keyword set comprises a structure keyword set and an appearance keyword set, structure text description and appearance text description are determined according to the structure keyword set, the appearance keyword set and a text description model, and the structure text description and the appearance text description are obtained according to a deep learning algorithm and the structure keyword set. Determining a ship structure line diagram, determining a ship reference image according to the appearance keyword set and the ship structure line diagram, and constructing a ship image data set according to the structure text description, the appearance text description and the ship reference image; according to the method, the ship image data set is expanded by adopting the generative model, and the ship identification accuracy is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, apparatus, device, and storage medium for constructing a vessel image dataset. Background Art

[0002] Vessel identification is a key link in the management process of the fishery industry. This link compares the current appearance of the vessel with its registered image to determine whether there are any problems such as duplicate license plates or illegal modifications.

[0003] At present, most of the methods used are manual verification, which requires a lot of manpower and material resources, is extremely costly, and the data collected is uneven. The types of ships collected are limited, the amount of data is far from enough, the efficiency is low, and the error rate is high, which greatly affects the effect of ship management.

[0004] However, due to the lack of publicly available large-scale ship datasets, some deep learning-based ship identification methods are limited by insufficient dataset size and data diversity, and are unable to achieve sufficient recognition rates, which are insufficient to completely replace manual review for widespread application. Summary of the Invention

[0005] The present application provides a method, apparatus, device and storage medium for constructing a vessel image dataset, which are used to solve the problem of insufficient number of vessel image datasets.

[0006] In a first aspect, the present application provides a method for constructing a vessel image dataset, comprising:

[0007] Acquire a keyword set of a vessel, the keyword set comprising: a structure keyword set and an appearance keyword set;

[0008] Determining a structural text description and an appearance text description according to the structural keyword set, the appearance keyword set, and a text description model;

[0009] Determining a ship structure line drawing based on a deep learning algorithm and the set of structural keywords;

[0010] determining a vessel reference image based on the appearance keyword set and the vessel structure line drawing;

[0011] A vessel image dataset is constructed according to the structural text description, the appearance text description, and the vessel reference image.

[0012] Optionally, determining the ship structure line drawing according to the deep learning algorithm and the structure keyword set includes:

[0013] Acquire an original ship image dataset, where the original ship image dataset is used to indicate an association relationship between the structure keyword set and the original ship image;

[0014] Acquire a target structure keyword set, wherein the target structure keyword set is composed of at least one keyword in the structure keyword set;

[0015] determining a target original ship image according to the target structure keyword set and the original ship image dataset;

[0016] Determining a target vessel structure line drawing based on the deep learning algorithm and the target original vessel image;

[0017] The ship structure line drawing is determined according to the target ship structure line drawing and the graphic video model.

[0018] Optionally, determining a vessel reference image based on the appearance keyword set and the vessel structure line drawing includes:

[0019] Inputting the appearance keyword set into a text encoder to generate appearance keyword text features;

[0020] Inputting the ship structure line drawing into an image encoder to generate ship structure image features;

[0021] determining a ship reference image feature value based on the appearance keyword text feature and the ship structure image feature;

[0022] According to the feature value of the ship reference image, a generation model is controlled to generate the ship reference image.

[0023] Optionally, inputting the appearance keyword set into a text encoder to generate appearance keyword text features includes:

[0024] Obtaining a target appearance keyword set, wherein the target appearance keyword set is composed of at least three keywords in the appearance keyword set;

[0025] Inputting the target appearance keyword set into the text encoder to generate a target appearance feature matrix set;

[0026] Determining a target appearance feature weight vector according to the target appearance feature matrix set;

[0027] The target appearance feature matrix set and the target appearance feature weight vector are used to determine the appearance keyword text feature.

[0028] Optionally, constructing a vessel image dataset according to the structural text description, the appearance text description, and the vessel reference image includes:

[0029] Obtain the ground truth image of the ship;

[0030] performing feature extraction on the true image of the ship according to the image encoder to determine feature values ​​of the true image;

[0031] determining a loss function value according to the true image feature value and the vessel reference image feature value;

[0032] When the loss function value is greater than a preset value, a ship image dataset is constructed according to the structure text description, the appearance text description, and the ship reference image.

[0033] Optionally, the appearance keyword text feature includes: a first appearance keyword text feature, a second appearance keyword text feature, and a third appearance keyword text feature, and determining the ship reference image feature value based on the appearance keyword text features and the ship structure image feature includes:

[0034] Performing a mutual attention mechanism on the first appearance keyword text feature and the vessel structure image feature to obtain reference image information;

[0035] Perform channel superposition processing on the second appearance keyword text feature and the third appearance keyword text feature to obtain text description information;

[0036] Determine a vessel reference image feature value according to the reference image information and the text description information.

[0037] Optionally, determining the ship reference image feature value based on the appearance keyword text feature and the ship structure image feature further includes:

[0038] The following formula is used to determine the characteristic value of the ship reference image:

[0039] f=cat(transformer(T α ,I r ),cat(T β ,λT γ ))

[0040] Among them, f is the characteristic value of the ship reference image, T α is the first appearance keyword text feature, T β Second appearance keyword text feature, T γ The third appearance keyword text feature, λ is the proportional coefficient, I r is the image feature of the ship structure.

[0041] In a second aspect, the present application provides a device for constructing a vessel image dataset, comprising:

[0042] An acquisition module, configured to acquire a keyword set of a vessel, wherein the keyword set includes a structure keyword set and an appearance keyword set;

[0043] a determination module, configured to determine a structural text description and an appearance text description according to the structural keyword set, the appearance keyword set, and a text description model;

[0044] The determining module is further configured to determine a ship structure line drawing based on a deep learning algorithm and the set of structural keywords;

[0045] The determining module is further configured to determine a vessel reference image based on the appearance keyword set and the vessel structure line drawing;

[0046] A processing module is used to construct a ship image dataset according to the structural text description, the appearance text description and the ship reference image.

[0047] Optionally, the acquisition module is further configured to acquire an original ship image dataset, wherein the original ship image dataset is used to indicate an association relationship between the structure keyword set and the original ship image;

[0048] The acquisition module is further configured to acquire a target structure keyword set, wherein the target structure keyword set is composed of at least one keyword in the structure keyword set;

[0049] The determining module is further configured to determine a target original ship image based on the target structure keyword set and the original ship image dataset;

[0050] The determination module is further configured to determine a target vessel structure line drawing based on the deep learning algorithm and the target original vessel image;

[0051] The determination module is further configured to determine the ship structure line drawing according to the target ship structure line drawing and the image-generated video model.

[0052] Optionally, the vessel image dataset construction device further includes: a generation module and a control module;

[0053] The generating module is configured to input the appearance keyword set into a text encoder to generate appearance keyword text features;

[0054] The generating module is further configured to input the ship structure line drawing into an image encoder to generate ship structure image features;

[0055] The determining module is further configured to determine a ship reference image feature value based on the appearance keyword text feature and the ship structure image feature;

[0056] The control module is used to control the generation model to generate the ship reference image according to the feature value of the ship reference image.

[0057] Optionally, the acquisition module is further configured to acquire a target appearance keyword set, wherein the target appearance keyword set is composed of at least three keywords in the appearance keyword set;

[0058] The generating module is further configured to input the target appearance keyword set into the text encoder to generate a target appearance feature matrix set;

[0059] The determination module is further configured to determine a target appearance feature weight vector based on the target appearance feature matrix set;

[0060] The determination module is further configured to determine the appearance keyword text feature by combining the target appearance feature matrix set and the target appearance feature weight vector.

[0061] Optionally, the acquisition module is further used to acquire a true image of the ship;

[0062] The determining module is further configured to extract features from the true image of the vessel using an image encoder to determine feature values ​​of the true image;

[0063] The determining module is further configured to determine a loss function value based on the true image feature value and the vessel reference image feature value;

[0064] The processing module is further configured to construct a vessel image dataset based on the structure text description, the appearance text description, and the vessel reference image when the loss function value is greater than a preset value.

[0065] Optionally, the processing module is further configured to perform a mutual attention mechanism on the first appearance keyword text feature and the vessel structure image feature to obtain reference image information;

[0066] The processing module is further configured to perform channel superposition processing on the second appearance keyword text feature and the third appearance keyword text feature to obtain text description information;

[0067] The determination module is further configured to determine a feature value of the vessel reference image based on the reference image information and the text description information.

[0068] Optionally, the determining module is further configured to determine the characteristic value of the vessel reference image using the following formula:

[0069] f=cat(transformer(T α ,I r ),cat(Tβ ,λT γ ))

[0070] Among them, f is the characteristic value of the ship reference image, T α is the first appearance keyword text feature, T β Second appearance keyword text feature, T γ The third appearance keyword text feature, λ is the proportional coefficient, I r is the image feature of the ship structure.

[0071] In a third aspect, the present application provides a device for constructing a vessel image dataset, comprising:

[0072] Memory;

[0073] processor;

[0074] wherein the memory stores computer-executable instructions;

[0075] The processor executes the computer-executable instructions stored in the memory to implement the method for constructing a vessel image dataset as described in the first aspect and various possible implementations of the first aspect and the second aspect.

[0076] In a fourth aspect, the present application provides a computer storage medium, characterized in that the computer storage medium stores computer execution instructions, and when the computer execution instructions are executed by a processor, they are used to implement the method for constructing a ship image dataset as described in the first aspect and various possible implementations of the first aspect.

[0077] The present application provides a method for constructing a ship image dataset, which obtains a keyword set for a ship, the keyword set including: a structural keyword set and an appearance keyword set; determines a structural text description and an appearance text description based on the structural keyword set, the appearance keyword set, and a text description model; determines a ship structure line drawing based on a deep learning algorithm and the structural keyword set; determines a ship reference image based on the appearance keyword set and the ship structure line drawing; and constructs a ship image dataset based on the structural text description, the appearance text description, and the ship reference image; the method uses a generative model to expand the ship image dataset and improve the accuracy of ship recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0079] Figure 1 Process of constructing the ship image dataset provided for this application Figure 1 ;

[0080] Figure 2 Process of constructing the ship image dataset provided for this application Figure 2 ;

[0081] Figure 3 Process of constructing the ship image dataset provided for this application Figure 3 ;

[0082] Figure 4 A schematic diagram of the structure of the device for constructing a ship image dataset provided in this application;

[0083] Figure 5 Schematic diagram of the structure of the device for constructing the ship image dataset provided in this application.

[0084] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0085] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0086] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0087] Vessel identification is a key link in the management process of the fishery industry. This link compares the current appearance of the vessel with its registered image to determine whether there are any problems such as duplicate license plates or illegal modifications.

[0088] At present, most of the methods used are manual verification, which requires a lot of manpower and material resources, is extremely costly, and the data collected is uneven. The types of ships collected are limited, the amount of data is far from enough, the efficiency is low, and the error rate is high, which greatly affects the effect of ship management.

[0089] However, due to the lack of publicly available large-scale ship datasets, some deep learning-based ship identification methods are limited by insufficient dataset size and data diversity, and are unable to achieve sufficient recognition rates, which are insufficient to completely replace manual review for widespread application.

[0090] In response to the above problems, the present application proposes a method for constructing a ship image dataset, which obtains a keyword set of ships including a structural keyword set and an appearance keyword set, determines a structural text description and an appearance text description based on the structural keyword set, the appearance keyword set and a text description model, determines a ship structure line drawing based on a deep learning algorithm and the structural keyword set, determines a ship reference image based on the appearance keyword set and the ship structure line drawing, and constructs a ship image dataset based on the structural text description, the appearance text description and the ship reference image; this method uses a generative model to expand the ship image dataset and improves the accuracy of ship recognition.

[0091] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0092] Figure 1 Process of constructing the ship image dataset provided for this application Figure 1 .like Figure 1 As shown, the method for constructing a vessel image dataset shown in this embodiment includes:

[0093] S101: Acquire a keyword set of a vessel, where the keyword set includes a structure keyword set and an appearance keyword set.

[0094] The keyword set of a ship is a vocabulary used to describe the characteristics of a ship, including a structure keyword set and an appearance keyword set.

[0095] The structural keyword set is used to describe the overall structure and components of the ship, such as: ship perspective, cabin front shape, cabin side shape, door location, number of front windows, number of side windows, and side window shape.

[0096] The appearance keyword set is used to describe the appearance features and details of the ship, such as: hull color, hull material, hatch color, hatch material, window frame color, window material, number of lights, light color, light position, lifebuoy position, radar position, weather, and background.

[0097] For example: the structural keyword set includes: ship perspective: {left, right, front, back}; shape of the front of the cabin: {square, rectangle, flat rectangle}; shape of the side of the cabin: {square, rectangle, flat rectangle, right trapezoid, isosceles trapezoid}; door position: {front of the cabin, back of the cabin, left side of the cabin, right side of the cabin}; number of front windows: {1, 2, 3, 4, 5, 6, 7}; number of side windows: {no window, 1, 2}; shape of side windows: {square, rectangle, parallelogram, circle}.

[0098] The appearance keyword set includes: hull color: {blue, white, black}; hull material: {metal, wood}; hatch door color: {blue, white, black, red}; hatch door material: {metal, wood}; window frame color: {blue, white, black, red, orange}; window material: {metal, wood, glass}; number of lights: {1, 2, 3, 4, 5}; light color: {white, black}; light position: {middle of the front of the cabin, two corners of the front of the cabin, middle of the side of the cabin, two corners of the side of the cabin}; lifebuoy position: {outside the cabin door, side of the cabin, top of the cabin}; radar position: {top of the cabin, left side of the cabin, right side of the cabin, back of the cabin}; weather: {sunny, rainy, foggy, cloudy, overcast, thunderstorm, hail, snow, sandstorm, floating dust, blowing sand, haze}; background: {sea surface, dock, river bank, city, mountain, woods}.

[0099] S102: Determine a structural text description and an appearance text description according to the structural keyword set, the appearance keyword set, and a text description model.

[0100] Among them, structural text description refers to a detailed text description of the structure, components and functions of the ship.

[0101] Appearance text description refers to a detailed text description of the vessel's appearance features, shape, and color.

[0102] First, a text description model is constructed that generates text describing the ship's structure and appearance based on a set of keywords. This model can be a rule-based template system or a machine learning-based natural language generation model. The structural keyword set is input into the text description model to generate a structural text description, while the appearance keyword set is input into the text description model to generate an appearance text description.

[0103] For example: Ship perspective: {left}; Cabin front shape: {square}; Cabin side shape: {square}; Door location: {cabin front}; Number of front windows: {3}; Number of side windows: {1}; Side window shape: {rectangular}. The structural text description corresponding to the structural keyword could be: For a ship viewed from the left, the cabin front shape is square, the cabin side shape is square, the door location is the cabin front, there are 3 front windows, there is 1 side window, and the side window shape is rectangular.

[0104] Hull color: {blue}; Hull material: {metal}; Hatch color: {blue}; Hatch material: {metal}; Window frame color: {blue}; Window material: {metal}; Number of lights: {1}; Light color: {white}; Light location: {center front of cabin}; Lifebuoy location: {outside cabin door}; Radar location: {top of cabin}; Weather: {clear}; Background: {sea surface}. The appearance text description corresponding to the appearance keyword could be: hull blue and made of metal; hatch blue and made of metal; window frame blue and made of metal; one white light in the center front of the cabin; lifebuoy outside the cabin door; radar on the top of the cabin; the weather was clear when the ship was photographed; the background was the sea surface.

[0105] S103: Determine a ship structure line drawing according to the deep learning algorithm and the structure keyword set.

[0106] Among them, the deep learning algorithm generates a line drawing of the ship structure based on a set of structural keywords.

[0107] A ship structure line drawing is a graphical representation of ship structures and components using lines and symbols.

[0108] You can select one or more deep learning algorithms, such as convolutional neural networks and recurrent neural networks, to learn and generate ship structure line drawings. Using the selected deep learning algorithm and training data, you can train a model capable of generating ship structure line drawings. During the training process, you can use a set of structural keywords as conditions or constraints to guide the model to generate line drawings that meet the requirements. The trained model is evaluated to check whether the generated line drawings meet the expected structural requirements. If the evaluation results are unsatisfactory, you can optimize the model or adjust its parameters to improve the generated quality.

[0109] S104: Determine a ship reference image according to the appearance keyword set and the ship structure line drawing.

[0110] The vessel reference image may be part of the algorithm output, which is generated by the algorithm after processing the original image.

[0111] The appearance keyword set is matched with the ship structure line drawing, the features described by the appearance keywords are combined with the ship structure line drawing, and a ship reference image that meets the requirements is generated based on the matching results.

[0112] S105: Construct a vessel image dataset according to the structural text description, the appearance text description, and the vessel reference image.

[0113] Among them, the ship image dataset is a collection of a large number of ship images. Its main uses include: ship recognition, identification, classification, tracking, etc., which can help researchers and developers better understand and analyze the behavior of ships in the marine environment.

[0114] As you can understand, we integrate structural text descriptions, appearance text descriptions, and reference images of ships into a ship image dataset, which is then split into training, validation, and test sets. The training set is used to train the model, the validation set is used to adjust model parameters and hyperparameters, and the test set is used to evaluate model performance. To increase the diversity of the ship image dataset and improve the generalization ability of the model, we can perform data augmentation on the images.

[0115] The present embodiment provides a method for constructing a ship image dataset, which obtains a keyword set for a ship, the keyword set including a structural keyword set and an appearance keyword set; determines a structural text description and an appearance text description based on the structural keyword set, the appearance keyword set, and a text description model; determines a ship structure line drawing based on a deep learning algorithm and the structural keyword set; determines a ship reference image based on the appearance keyword set and the ship structure line drawing; and constructs a ship image dataset based on the structural text description, the appearance text description, and the ship reference image. This method uses a generative model to expand the ship image dataset and improve the accuracy of ship recognition.

[0116] Figure 2 Process of constructing the ship image dataset provided for this application Figure 2 .like Figure 2 As shown, this embodiment is in Figure 1 Based on the embodiment, a method for constructing a vessel image dataset is described in detail. The method for constructing a vessel image dataset shown in this embodiment includes:

[0117] S201: Acquire a keyword set of a vessel, where the keyword set includes a structure keyword set and an appearance keyword set.

[0118] Step S201 is similar to step S101 and will not be described again here.

[0119] S202: Determine a structural text description and an appearance text description according to the structural keyword set, the appearance keyword set, and a text description model.

[0120] Among them, step S202 is similar to step S102 and will not be repeated here.

[0121] S203: Acquire an original ship image dataset, where the original ship image dataset is used to indicate an association relationship between the structure keyword set and the original ship image.

[0122] Among them, the original ship image dataset usually refers to a collection of unprocessed, real ship images, which may come from various sensors, cameras or satellites.

[0123] Many research institutions have distributed public ship image datasets for researchers and developers to use. For example, datasets such as DIOR, LEVIR, VEDAI, and HRSIO all include a large number of ship images.

[0124] The original ship image dataset is characterized by a small amount of data and a small number of original ship images. Therefore, the accuracy of the results determined in the ship recognition process is not enough. The original ship image dataset can be expanded as a basic dataset to train a richer and more diverse ship image dataset and improve the accuracy of ship recognition.

[0125] S204: Acquire a target structure keyword set, where the target structure keyword set is used to indicate any subset of the structure keyword set.

[0126] The structural keyword set is composed of structural keyword subsets at multiple levels, and the target structural keyword set is composed of one keyword selected from the structural keyword subsets at each level.

[0127] For example: the structural keyword set is: ship perspective: {left, right, front, back}; shape of the front of the cabin: {square, rectangle, flat rectangle}; shape of the side of the cabin: {square, rectangle, flat rectangle, right trapezoid, isosceles trapezoid}; door position: {front of the cabin, back of the cabin, left side of the cabin, right side of the cabin}; number of front windows: {1, 2, 3, 4, 5, 6, 7}; number of side windows: {no window, 1, 2}; shape of side windows: {square, rectangle, parallelogram, circle}.

[0128] Structural keywords include: ship perspective, cabin front shape, cabin side shape, door position, number of front windows, number of side windows, and side window shape.

[0129] The target structure keyword set can be: ship perspective: {left}; cabin front shape: {square}; cabin side shape: {square}; door position: {cabin front}; number of front windows: {3}; number of side windows: {1}; side window shape: {rectangle}.

[0130] S205: Determine a target original ship image according to the target structure keyword set and the original ship image dataset.

[0131] It can be understood that a structure keyword set that meets the target structure keyword set is selected from the original ship image dataset, and an original ship image corresponding to the structure keyword set in the original ship image dataset is selected and used as the target original ship image.

[0132] For example, the target structure keyword set may be: ship perspective: {left}; cabin front shape: {square}; cabin side shape: {square}; cabin door position: {cabin front}; number of front windows: {3}; number of side windows: {1}; side window shape: {rectangle}.

[0133] The original ship image dataset includes: structural keyword set - ship perspective: {left}; cabin front shape: {square}; cabin side shape: {square}; door position: {cabin front}; number of front windows: {3}; number of side windows: {1}; side window shape: {rectangle} corresponding to the original ship images - Image 1, Image 2, Image 3 and Image 4.

[0134] The target original ship images are: Image 1, Image 2, Image 3, and Image 4.

[0135] S206: Determine a target ship structure line drawing based on the deep learning algorithm and the target original ship image.

[0136] Among them, deep learning algorithms are designed for tasks such as image segmentation or image generation, and U-Net and GAN can be used.

[0137] The original target ship image can be preprocessed first, including image scaling and normalization, to ensure that the deep learning algorithm can accurately generate the target ship structure line drawing. The generated target ship structure line drawing can then be post-processed, such as smoothing lines and removing noise, to improve the quality of the target ship structure line drawing.

[0138] S207: Determine the target ship structure line drawing according to the target ship structure line drawing and the image-generated video model.

[0139] Among them, the target ship structure line diagram includes the ship rigid body structure line diagram from several perspectives. Through the image-generated video model, the target ship structure line diagram can be combined in multiple dimensions to determine the ship's structure line diagram from all angles. The ship's structure line diagram from all angles is the ship's structure line diagram.

[0140] S208: Input the appearance keyword set into a text encoder to generate appearance keyword text features.

[0141] Among them, choose a suitable text encoder, for example: an encoder based on a recurrent neural network (RNN), such as: long short-term memory (LSTM) or gated recurrent unit (GRU), or an encoder based on a self-attention mechanism, such as: Transformer.

[0142] The appearance keyword set is preprocessed, including word segmentation, stop word removal, stemming, or part-of-speech restoration. The preprocessed appearance keyword sequence is input into a text encoder, which converts the keyword sequence into a fixed-length text feature vector. The text features of the appearance keywords are extracted from the output of the text encoder to obtain the appearance keyword text features.

[0143] Optionally, the specific implementation method of this step is:

[0144] Obtaining a target appearance keyword set, wherein the target appearance keyword set is composed of at least three keywords in the appearance keyword set;

[0145] Inputting the target appearance keyword set into the text encoder to generate a target appearance feature matrix set;

[0146] Determining a target appearance feature weight vector according to the target appearance feature matrix set;

[0147] The target appearance feature matrix set and the target appearance feature weight vector are used to determine the appearance keyword text feature.

[0148] S209: Inputting the ship structure line drawing into an image encoder to generate ship structure image features.

[0149] Among them, select a suitable image encoder, such as a convolutional neural network (CNN), such as VGG, ResNet, Inception, etc.

[0150] The ship structure line drawing is preprocessed, including image scaling, normalization, and denoising, to ensure that the image quality is suitable for input into the image editor. The preprocessed ship structure line drawing is input into the image encoder, which converts the image into a series of feature maps. At the end of the image encoder, there is usually a fully connected layer or average pooling layer to encode the feature map into a fixed-length feature vector. The extracted features can be further processed, such as normalization and PAC dimensionality reduction, to obtain the ship structure image features.

[0151] S210: Determine a ship reference image feature value according to the appearance keyword text feature and the ship structure image feature.

[0152] The ship reference image feature value is a numerical value or vector extracted and calculated from the ship reference image, and is used to describe and distinguish the characteristics and attributes of the ship.

[0153] Optionally, a specific implementation of this step is: the appearance keyword text feature includes: a first appearance keyword text feature, a second appearance keyword text feature, and a third appearance keyword text feature.

[0154] Performing a mutual attention mechanism on the first appearance keyword text feature and the vessel structure image feature to obtain reference image information;

[0155] Perform channel superposition processing on the second appearance keyword text feature and the third appearance keyword text feature to obtain text description information;

[0156] Determine a vessel reference image feature value according to the reference image information and the text description information.

[0157] The following formula is used to determine the characteristic value of the ship reference image:

[0158] f=cat(transformer(T α ,I r ),cat(T β ,λT γ ))

[0159] Among them, f is the characteristic value of the ship reference image, T α is the first appearance keyword text feature, T β Second appearance keyword text feature, T γ The third appearance keyword text feature, λ is the proportional coefficient, I r is the image feature of the ship structure.

[0160] It can be understood that if the appearance keyword set has N keyword sets, then N corresponding text feature sets are obtained. The number of keyword sets in the three parts is recorded as p, q, and r respectively, then p+q+r=N.

[0161] Assume that the text features generated by all keywords in each set of the three parts constitute a feature matrix, which is recorded as α = [α1, α2, ..., α p ]、β=[β1,β2,…,β q ]、γ=[γ1,γ2,…,γ r ].

[0162] Set a weight vector for each set. The length of the vector is the same as the number of elements in the set. In each weight vector, only one element is 1, and the rest are 0. The weight vectors corresponding to all sets in the three parts constitute the weight matrix, which is denoted as w i =[w i1 ,w i2 ,…,w ip ]、w m =[w m1 ,w m2 ,…,w mq ]、w b =[w b1 ,w b2 ,…,w br ].

[0163] Since in the second part, not every keyword set needs to select keywords, a weight vector w is designed for it β , whose length is the same as the number of sets in the second part, that is, w β ∈R q , whose elements are 0 or 1, 0 means that the keyword set is not selected, and 1 means that the keyword set is selected.

[0164] Randomly initialize weights w β 、w i 、w m 、w b , that is, the random selection of keywords is completed. The text features of the three parts selected can be written as follows:

[0165] T α =[w i1 *α1,w i2 *α2,…,w ip *α p ]=w i α

[0166] T β =w β *[w m1 *β1,wm2 *β2,…,w mq *β q ]=w β w m β

[0167] T γ =[w b1 *γ1,w b2 *γ2,…,w br *γ r ]=w b γ

[0168] For example: The first appearance keyword text features include: hull color: {blue, white, black}; hull material: {metal, wood}; hatch color: {blue, white, black, red}; hatch material: {metal, wood}; window frame color: {blue, white, black, red, orange}; window material: {metal, wood, glass}.

[0169] The second appearance keyword text features include: number of lights: {1, 2, 3, 4, 5}; light color: {white, black}; light position: {middle of the front of the cabin, two corners of the front of the cabin, middle of the side of the cabin, two corners of the side of the cabin}; lifebuoy position: {outside the cabin door, side of the cabin, top of the cabin}; radar position: {top of the cabin, left side of the cabin, right side of the cabin, back of the cabin}.

[0170] The third appearance keyword text features include: weather: {sunny, rainy, foggy, cloudy, overcast, thunderstorm, hail, snow, sandstorm, dust, blowing sand, haze}; background: {sea surface, dock, river bank, city, mountain, forest}.

[0171] S211: Controlling a generation model to generate the ship reference image according to the feature value of the ship reference image.

[0172] The generative model can be a generative adversarial network (GAN). A GAN is an image generation technique that consists of two main components: a generator and a discriminator. The generator's task is to generate images that are as realistic as possible to deceive the discriminator, while the discriminator's task is to distinguish between generated images and real images.

[0173] The feature values ​​of the ship reference image can be input into the generative model, and quantitative and qualitative evaluation methods can be used to verify whether the generated image meets expectations, and the image that meets expectations can be used as the ship reference image.

[0174] S212: Construct a vessel image dataset according to the structural text description, the appearance text description, and the vessel reference image.

[0175] Among them, step S212 is similar to step S105 and will not be repeated here.

[0176] The present embodiment provides a method for constructing a ship image dataset, which obtains a keyword set for a ship, the keyword set including a structure keyword set and an appearance keyword set; determines a structure text description and an appearance text description based on the structure keyword set, the appearance keyword set, and a text description model; obtains an original ship image dataset; obtains a target structure keyword set corresponding to any subset of the structure keyword set; determines a target original ship image based on the target structure keyword set and the original ship image dataset; determines a target ship structure line drawing based on a deep learning algorithm and the target original ship image; determines a ship structure line drawing based on the target ship structure line drawing and an image-generated video model; inputs the appearance keyword set into a text encoder to generate appearance keyword text features; inputs the ship structure line drawing into an image encoder to generate ship structure image features; determines a ship reference image feature value based on the appearance keyword text features and the ship structure image features; controls a generative model to generate the ship reference image based on the ship reference image feature values; and constructs a ship image dataset based on the structure text description, the appearance text description, and the ship reference image. The method expands the ship image dataset using the generative model and improves the accuracy of ship recognition.

[0177] Figure 3 Process of constructing the ship image dataset provided for this application Figure 3 .like Figure 3 As shown, this embodiment is in Figure 2 Based on the embodiment, a detailed description is given of constructing a vessel image dataset based on the structure text description, the appearance text description, and the vessel reference image. The method for constructing the vessel image dataset shown in this embodiment includes:

[0178] S301: Acquire a ground-truth image of a vessel.

[0179] Acquiring ground-truth vessel images typically refers to obtaining true annotated images corresponding to the image being processed, such as in tasks like vessel recognition, identification, or segmentation. These ground-truth annotated images typically contain the precise location and bounding boxes of the vessel, pixel-level segmentation masks, or other forms of vessel location information. Ground-truth images are used for training and evaluating models to ensure they can accurately identify and locate vessels.

[0180] S302: performing feature extraction on the true image of the ship according to the image encoder to determine feature values ​​of the true image.

[0181] The image encoder is typically a pre-trained deep learning model. For ground-truth images of ships, choose a model pre-trained on similar image data, such as one pre-trained on ImageNet. Common image encoders include VGG, ResNet, and MobileNet.

[0182] Perform necessary preprocessing on the ground-truth ship images to make them suitable for the input requirements of the image encoder. This may include resizing the images, normalizing pixel values ​​(typically scaling pixel values ​​to the range of [0, 1] or [-1, 1]), and data augmentation (such as random cropping and rotation).

[0183] Load the pre-trained image encoder model and make sure it can accept pre-processed images as input. If you are using a deep learning framework (such as TensorFlow or PyTorch), you can use the corresponding API to load the pre-trained model.

[0184] The preprocessed ground-truth image is fed into the image encoder. The image encoder extracts image features through a series of convolutional and pooling layers. During feature extraction, you can choose to extract features at different layers. Generally speaking, shallower features capture local information, while deeper features capture more global and abstract information. This determines the ground-truth image feature values.

[0185] S303: Determine a loss function value according to the true image feature value and the ship reference image feature value.

[0186] Among them, the loss function can use cosine similarity loss.

[0187] Cosine similarity measures the angle between two vectors, and its value range is [-1, 1], where 1 indicates the exact same direction and -1 indicates the exact opposite direction.

[0188] The cosine similarity loss can be defined as:

[0189]

[0190] Among them, I gt is the true image eigenvalue, and f is the ship reference image eigenvalue.

[0191] S304: When the loss function value is greater than a preset value, constructing a ship image dataset according to the structure text description, the appearance text description, and the ship reference image.

[0192] When the loss function value is greater than the preset value, it indicates that the model has a small error when generating the reference ship image based on the ground-truth image feature values. In this case, a ship image dataset can be constructed by combining the structural text description, the appearance text description, and the reference ship images for further training or optimization of the model.

[0193] The present embodiment provides a method for constructing a ship image dataset, which obtains a true image of a ship, performs feature extraction on the true image of the ship using an image encoder, determines a true image feature value, and determines a loss function value based on the true image feature value and the feature value of the ship reference image. When the loss function value is greater than a preset value, a ship image dataset is constructed based on the structural text description, the appearance text description, and the ship reference image. This method uses a generative model to expand the ship image dataset and improve the accuracy of ship recognition.

[0194] Figure 4 Schematic diagram of the structure of the vessel image dataset construction device provided in this application Figure 2 .like Figure 4 As shown, the vessel image dataset construction device 400 provided in this embodiment includes:

[0195] An acquisition module 401 is used to acquire a keyword set of a vessel, wherein the keyword set includes a structure keyword set and an appearance keyword set;

[0196] A determination module 402 is configured to determine a structural text description and an appearance text description based on the structural keyword set, the appearance keyword set, and a text description model;

[0197] The determining module 402 is further configured to determine a ship structure line drawing based on a deep learning algorithm and the structure keyword set;

[0198] The determining module 402 is further configured to determine a vessel reference image based on the appearance keyword set and the vessel structure line drawing;

[0199] The processing module 403 is configured to construct a vessel image dataset according to the structural text description, the appearance text description, and the vessel reference image.

[0200] Optionally, the acquisition module 401 is further configured to acquire an original ship image dataset, wherein the original ship image dataset is used to indicate an association relationship between the structure keyword set and the original ship image;

[0201] The acquisition module 401 is further configured to acquire a target structure keyword set, wherein the target structure keyword set is composed of at least one keyword in the structure keyword set;

[0202] The determining module 402 is further configured to determine a target original ship image based on the target structure keyword set and the original ship image dataset;

[0203] The determining module 402 is further configured to determine a target vessel structure line drawing based on the deep learning algorithm and the target original vessel image;

[0204] The determining module 402 is further configured to determine the ship structure line drawing according to the target ship structure line drawing and the image-generated video model.

[0205] Optionally, the vessel image dataset construction device further includes: a generation module 404 and a control module 405;

[0206] The generating module 404 is configured to input the appearance keyword set into a text encoder to generate appearance keyword text features;

[0207] The generating module 404 is further configured to input the ship structure line drawing into an image encoder to generate ship structure image features;

[0208] The determining module 402 is further configured to determine a feature value of a ship reference image based on the appearance keyword text feature and the ship structure image feature;

[0209] The control module 405 is configured to control a generation model to generate the ship reference image according to the feature value of the ship reference image.

[0210] Optionally, the acquisition module 401 is further configured to acquire a target appearance keyword set, wherein the target appearance keyword set is composed of at least three keywords in the appearance keyword set;

[0211] The generating module 404 is further configured to input the target appearance keyword set into the text encoder to generate a target appearance feature matrix set;

[0212] The determining module 402 is further configured to determine a target appearance feature weight vector based on the target appearance feature matrix set;

[0213] The determining module 402 is further configured to determine the appearance keyword text feature by combining the target appearance feature matrix set and the target appearance feature weight vector.

[0214] Optionally, the acquisition module 401 is further configured to acquire a true image of the vessel;

[0215] The determining module 402 is further configured to extract features from the true image of the vessel using an image encoder to determine feature values ​​of the true image;

[0216] The determining module 402 is further configured to determine a loss function value based on the true image feature value and the vessel reference image feature value;

[0217] The processing module 403 is further configured to construct a ship image dataset according to the structure text description, the appearance text description, and the ship reference image when the loss function value is greater than a preset value.

[0218] Optionally, the processing module 403 is further configured to perform a mutual attention mechanism on the first appearance keyword text feature and the vessel structure image feature to obtain reference image information;

[0219] The processing module 403 is further configured to perform channel superposition processing on the second appearance keyword text feature and the third appearance keyword text feature to obtain text description information;

[0220] The determining module 402 is further configured to determine a feature value of the vessel reference image according to the reference image information and the text description information.

[0221] Optionally, the determining module 402 is further configured to determine the characteristic value of the vessel reference image using the following formula:

[0222] f=cat(transformer(T α ,I r ),cat(T β ,λT γ ))

[0223] Among them, f is the characteristic value of the ship reference image, T α is the first appearance keyword text feature, T β Second appearance keyword text feature, T γ The third appearance keyword text feature, λ is the proportional coefficient, I r is the image feature of the ship structure.

[0224] Figure 5 Schematic diagram of the structure of the equipment for constructing the ship image dataset provided in this application. Figure 5 As shown, the present application provides a vessel image dataset construction device, and the vessel image dataset construction device 500 includes: a receiver 501 , a transmitter 502 , a processor 503 and a memory 504 .

[0225] Receiver 501, for receiving instructions and data;

[0226] Transmitter 502, used to send instructions and data;

[0227] Memory 504, for storing computer-executable instructions;

[0228] The processor 503 is configured to execute the computer-executable instructions stored in the memory 504 to implement the various steps of the method for constructing a vessel image dataset in the above embodiment. For details, please refer to the relevant description of the embodiment of the method for constructing a vessel image dataset.

[0229] Optionally, the memory 504 may be independent or integrated with the processor 503 .

[0230] When the memory 504 is independently provided, the electronic device further includes a bus for connecting the memory 504 and the processor 503 .

[0231] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the ship image dataset construction method executed by the above-mentioned ship image dataset construction device is implemented.

[0232] It should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all optional embodiments, and the actions and modules involved are not necessarily required by this application.

[0233] It should be further noted that, although the various steps in the flowchart are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps may be performed in other orders. Moreover, at least a portion of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily performed at the same time, but may be performed at different times. The execution order of these sub-steps or stages is not necessarily to be performed in sequence, but may be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0234] It should be understood that the above-described device embodiments are merely illustrative, and the device of the present application may also be implemented in other ways. For example, the division of units / modules in the above-described embodiments is merely a logical functional division, and actual implementations may employ other division methods. For example, multiple units, modules, or components may be combined or integrated into another system, or some features may be omitted or not implemented.

[0235] In addition, unless otherwise specified, the functional units / modules in the various embodiments of the present application may be integrated into a single unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The aforementioned integrated units / modules may be implemented in the form of hardware or software program modules.

[0236] If the integrated unit / module is implemented in hardware, the hardware may be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor may be any appropriate hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC. Unless otherwise specified, the storage unit may be any appropriate magnetic storage medium or magneto-optical storage medium, such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), etc.

[0237] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes a number of instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned memory includes various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0238] In the above embodiments, the description of each embodiment has its own focus. For parts not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0239] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of the present application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0240] It should be understood that the present application is not limited to the exact structure described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for constructing a ship image dataset, characterized in that: The method comprises: Acquire a keyword set of a vessel, the keyword set comprising: a structure keyword set and an appearance keyword set; Determining a structural text description and an appearance text description according to the structural keyword set, the appearance keyword set, and a text description model; Determining a ship structure line drawing based on a deep learning algorithm and the set of structural keywords; determining a vessel reference image based on the appearance keyword set and the vessel structure line drawing; A vessel image dataset is constructed according to the structural text description, the appearance text description, and the vessel reference image.

2. The method according to claim 1, characterized in that The determining of the ship structure line drawing according to the deep learning algorithm and the structure keyword set includes: Acquire an original ship image dataset, where the original ship image dataset is used to indicate an association relationship between the structure keyword set and the original ship image; Acquire a target structure keyword set, wherein the target structure keyword set is composed of at least one keyword in the structure keyword set; determining a target original ship image according to the target structure keyword set and the original ship image dataset; Determining a target vessel structure line drawing based on the deep learning algorithm and the target original vessel image; The ship structure line drawing is determined according to the target ship structure line drawing and the graphic video model.

3. The method according to claim 1, characterized in that Determining a vessel reference image according to the appearance keyword set and the vessel structure line drawing includes: Inputting the appearance keyword set into a text encoder to generate appearance keyword text features; Inputting the ship structure line drawing into an image encoder to generate ship structure image features; determining a ship reference image feature value based on the appearance keyword text feature and the ship structure image feature; According to the feature value of the ship reference image, a generation model is controlled to generate the ship reference image.

4. The method according to claim 3, characterized in that The step of inputting the appearance keyword set into a text encoder to generate appearance keyword text features includes: Obtaining a target appearance keyword set, wherein the target appearance keyword set is composed of at least three keywords in the appearance keyword set; Inputting the target appearance keyword set into the text encoder to generate a target appearance feature matrix set; Determining a target appearance feature weight vector according to the target appearance feature matrix set; The target appearance feature matrix set and the target appearance feature weight vector are used to determine the appearance keyword text feature.

5. The method according to claim 3, characterized in that The constructing of a vessel image dataset according to the structure text description, the appearance text description, and the vessel reference image includes: Obtain the ground truth image of the ship; performing feature extraction on the true image of the ship according to the image encoder to determine feature values ​​of the true image; determining a loss function value according to the true image feature value and the vessel reference image feature value; When the loss function value is greater than a preset value, a ship image dataset is constructed according to the structure text description, the appearance text description, and the ship reference image.

6. The method according to claim 3, characterized in that The appearance keyword text features include: a first appearance keyword text feature, a second appearance keyword text feature, and a third appearance keyword text feature. Determining the ship reference image feature value based on the appearance keyword text features and the ship structure image features includes: Performing a mutual attention mechanism on the first appearance keyword text feature and the vessel structure image feature to obtain reference image information; Perform channel superposition processing on the second appearance keyword text feature and the third appearance keyword text feature to obtain text description information; Determine a vessel reference image feature value according to the reference image information and the text description information.

7. The method according to claim 6, characterized in that The determining of the ship reference image feature value according to the appearance keyword text feature and the ship structure image feature further includes: The following formula is used to determine the characteristic value of the ship reference image: f=cat(transformer(T α ,I r ),cat(T β ,λT γ )) Among them, f is the characteristic value of the ship reference image, T α is the first appearance keyword text feature, T β Second appearance keyword text feature, T γ The third appearance keyword text feature, λ is the proportional coefficient, I r is the image feature of the ship structure.

8. A device for constructing a vessel image dataset, characterized in that: include: An acquisition module, configured to acquire a keyword set of a vessel, wherein the keyword set includes a structure keyword set and an appearance keyword set; a determination module, configured to determine a structural text description and an appearance text description according to the structural keyword set, the appearance keyword set, and a text description model; The determining module is further configured to determine a ship structure line drawing based on a deep learning algorithm and the structure keyword set; The determining module is further configured to determine a vessel reference image based on the appearance keyword set and the vessel structure line drawing; A processing module is used to construct a ship image dataset according to the structural text description, the appearance text description and the ship reference image.

9. A device for constructing a vessel image dataset, characterized in that: include: a processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images