Ship emergency feedback method and system based on image and text joint analysis

Abstract

The application discloses a ship emergency feedback method and system based on image and text joint analysis, and the method comprises the following steps: acquiring an emergency disposal text sent by a ship end; identifying a plurality of position keywords in the emergency disposal text based on a text character recognition algorithm; determining corresponding position associated images in historical image data of the ship end based on the position keywords; and generating a standardized feedback record corresponding to the emergency disposal text according to joint analysis of the position associated images and the emergency disposal text. It can be seen that the application can realize intelligent position association and standardized processing of emergency disposal information, improve the accuracy and standardization of ship emergency feedback, and reduce the risk of feedback delay or information loss caused by manual recording.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a ship emergency feedback method and system based on joint analysis of images and text. Background Technology

[0002] With the rapid growth in demand for intelligent emergency management and safety supervision of ships, shipping companies and ship operators are increasingly emphasizing the improvement of feedback accuracy and standardization through intelligent processing of emergency response information. A key technical issue is how to automate the processing of emergency response texts to reduce delays and information gaps caused by manual recording. Existing technologies typically acquire emergency response texts sent from the ship, record the information manually or through simple keyword matching, and generate emergency feedback records based on manual descriptions. However, existing solutions lack precise extraction of location keywords using text character recognition algorithms, automatic correlation analysis of historical image data, and a standardized feedback record generation mechanism driven by joint analysis. This makes it difficult to achieve intelligent location correlation and standardized processing of emergency response information, resulting in insufficient feedback accuracy and standardization. Furthermore, delays or information gaps due to manual recording can easily lead to low emergency response efficiency, limiting the intelligence level and responsiveness of ship emergency management systems. Therefore, existing technologies have shortcomings that urgently need to be addressed. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a ship emergency feedback method and system based on joint analysis of images and text, which can realize intelligent location association and standardized processing of emergency response information, improve the accuracy and standardization of ship emergency feedback, and reduce the risk of feedback delay or information loss caused by manual recording.

[0004] To address the aforementioned technical problems, the first aspect of this invention discloses a ship emergency feedback method based on joint image and text analysis, the method comprising: Obtain emergency response text sent from the ship; Based on a text character recognition algorithm, multiple location keywords in the emergency response text are identified; Based on the location keywords, the corresponding location-related images are determined from the historical image data uploaded by the ship. Based on the joint analysis of the location-related image and the emergency response text, a standardized feedback record corresponding to the emergency response text is generated.

[0005] As an optional implementation, in the first aspect of the present invention, the identification of multiple location keywords in the emergency response text based on the text character recognition algorithm includes: Based on the LLM model, multiple possible location representation words corresponding to the ship end are generated; Based on the word segmentation algorithm, the emergency response text is divided into multiple text words; For each of the text words, calculate the average similarity between the text word and each of the possible positional character words to obtain the similarity parameter corresponding to the text word; Based on the similarity parameters, multiple location keywords are selected from all the text words.

[0006] As an optional implementation, in the first aspect of the present invention, the generation of multiple possible location representation words corresponding to the ship end based on the LLM model includes: Obtain the ship parameters and ship description corresponding to the ship terminal; the ship parameters include at least one of ship model, ship personnel information, ship hardware information, and ship cargo information; Based on preset location description prompt rules, the ship parameters and ship description are used to generate corresponding location description prompts; The location description prompts are input into the trained LLM model to obtain multiple output location descriptions; the LLM model is trained using a training dataset that includes multiple training ship data and corresponding ship location description annotations. Calculate the repetition degree corresponding to each of the location descriptors; the repetition degree is the average of the similarity between each of the location descriptors and all other location descriptors; The location descriptors with a repetition degree less than a preset repetition degree threshold are selected to obtain multiple possible location characterization words.

[0007] As an optional implementation, in the first aspect of the invention, the step of filtering multiple location keywords from all the text words based on the similarity parameter includes: Determine whether the similarity parameter corresponding to the text word is greater than a preset first parameter threshold, and obtain the first determination result; If the first determination result is yes, the text word is identified as a location keyword; If the first judgment result is negative, it is determined whether the similarity parameter corresponding to the text word is less than the preset second parameter threshold to obtain the second judgment result; the second parameter threshold is less than the first parameter threshold. If the second judgment result is yes, the text word will not be identified as a location keyword; If the second judgment result is negative, determine whether the similarity parameter corresponding to the text word falls within the corresponding reference similarity parameter range to obtain the third judgment result; When the third judgment result is yes, the text word is identified as a location keyword; If the third judgment result is negative, the text word will not be identified as a location keyword.

[0008] As an optional implementation, in the first aspect of the invention, the reference similarity parameter range is calculated in the following manner: Text words whose similarity parameters are greater than the first parameter threshold are identified as reference text words; Calculate the average value of the similarity parameters corresponding to all the reference text words to obtain the median of the interval; Calculate the standard deviation of the similarity parameter corresponding to all the reference text words to obtain the interval representation value; Based on the total number of all the reference text words and the pre-defined correspondence between the number and the deviation parameter, the corresponding interval deviation parameter is determined; The interval deviation value is obtained by multiplying the interval deviation parameter and the interval characterization value. The lower limit of the interval is determined by subtracting the interval deviation value from the interval median value, and the upper limit of the interval is determined by adding the interval deviation value to the interval median value, so as to obtain the reference similarity parameter interval.

[0009] As an optional implementation, in the first aspect of the present invention, determining the corresponding location-related image from the historically uploaded image data at the ship end based on the location keywords includes: For each location keyword, the location coordinates corresponding to the location keyword are predicted based on the coordinate prediction model corresponding to the ship end. For each image data historically uploaded from the ship, calculate the text similarity between the location keyword and the location text tag of the image data; Calculate the positional distance between the positional coordinates corresponding to the location keyword and the positional coordinates label of the image data; Calculate the weighted sum of the text similarity and the location distance to obtain the matching degree between the location keyword and the image data; The image data with the highest matching degree is determined as the location-related image corresponding to the location keyword.

[0010] As an optional implementation, in the first aspect of the invention, generating a standardized feedback record corresponding to the emergency response text based on the joint analysis of the location-related image and the emergency response text includes: Determine the type of emergency event corresponding to the emergency response text; Select the image recognition model corresponding to the emergency event type from the preset image recognition model library; Each location-associated image is input into the image recognition model to obtain the emergency response recognition result; the emergency response recognition result includes the emergency scene image area, the emergency response image area, and the degree of emergency response completion. The location-associated image and the corresponding emergency response identification result are filled into the standardized feedback template corresponding to the emergency event type to obtain the corresponding standardized feedback record.

[0011] As an optional implementation, in the first aspect of the present invention, determining the emergency event type corresponding to the emergency response text includes: Determine the text location corresponding to each of the aforementioned location keywords in the emergency response text; Based on the text location, all the location keywords are clustered to obtain a keyword set; the keyword set includes multiple location keywords whose text locations are less than a preset distance threshold. The shortest text segment in the emergency response text that includes all the location keywords in the keyword set is identified as the text to be predicted; The text to be predicted is input into a trained emergency event type prediction model to obtain the emergency event type corresponding to the emergency response text; the emergency event type prediction model is trained using a training dataset that includes multiple training emergency text descriptions and corresponding emergency event type annotations.

[0012] A second aspect of this invention discloses a ship emergency feedback system based on joint image and text analysis, the system comprising: The acquisition module is used to acquire emergency response texts sent from the ship. The recognition module is used to identify multiple location keywords in the emergency response text based on a text character recognition algorithm; The determination module is used to determine the corresponding location-related image in the historical image data uploaded by the ship based on the location keywords; The generation module is used to generate a standardized feedback record corresponding to the emergency response text based on the joint analysis of the location-related image and the emergency response text.

[0013] As an optional implementation, in a second aspect of the invention, the specific method by which the recognition module identifies multiple location keywords in the emergency response text based on a text character recognition algorithm includes: Based on the LLM model, multiple possible location representation words corresponding to the ship end are generated; Based on the word segmentation algorithm, the emergency response text is divided into multiple text words; For each of the text words, calculate the average similarity between the text word and each of the possible positional character words to obtain the similarity parameter corresponding to the text word; Based on the similarity parameters, multiple location keywords are selected from all the text words.

[0014] As an optional implementation, in the second aspect of the present invention, the specific method by which the identification module generates multiple possible location representation words corresponding to the ship end based on an LLM model includes: Obtain the ship parameters and ship description corresponding to the ship terminal; the ship parameters include at least one of ship model, ship personnel information, ship hardware information, and ship cargo information; Based on preset location description prompt rules, the ship parameters and ship description are used to generate corresponding location description prompts; The location description prompts are input into the trained LLM model to obtain multiple output location descriptions; the LLM model is trained using a training dataset that includes multiple training ship data and corresponding ship location description annotations. Calculate the repetition degree corresponding to each of the location descriptors; the repetition degree is the average of the similarity between each of the location descriptors and all other location descriptors; The location descriptors with a repetition degree less than a preset repetition degree threshold are selected to obtain multiple possible location characterization words.

[0015] As an optional implementation, in a second aspect of the invention, the specific method by which the identification module filters out multiple location keywords from all the text words based on the similarity parameters includes: Determine whether the similarity parameter corresponding to the text word is greater than a preset first parameter threshold, and obtain the first determination result; If the first determination result is yes, the text word is identified as a location keyword; If the first judgment result is negative, it is determined whether the similarity parameter corresponding to the text word is less than the preset second parameter threshold to obtain the second judgment result; the second parameter threshold is less than the first parameter threshold. If the second judgment result is yes, the text word will not be identified as a location keyword; If the second judgment result is negative, determine whether the similarity parameter corresponding to the text word falls within the corresponding reference similarity parameter range to obtain the third judgment result; When the third judgment result is yes, the text word is identified as a location keyword; If the third judgment result is negative, the text word will not be identified as a location keyword.

[0016] As an optional implementation, in a second aspect of the invention, the reference similarity parameter range is calculated in the following manner: Text words whose similarity parameters are greater than the first parameter threshold are identified as reference text words; Calculate the average value of the similarity parameters corresponding to all the reference text words to obtain the median of the interval; Calculate the standard deviation of the similarity parameter corresponding to all the reference text words to obtain the interval representation value; Based on the total number of all the reference text words and the pre-defined correspondence between the number and the deviation parameter, the corresponding interval deviation parameter is determined; The interval deviation value is obtained by multiplying the interval deviation parameter and the interval characterization value. The lower limit of the interval is determined by subtracting the interval deviation value from the interval median value, and the upper limit of the interval is determined by adding the interval deviation value to the interval median value, so as to obtain the reference similarity parameter interval.

[0017] As an optional implementation, in a second aspect of the invention, the specific method by which the determining module determines the corresponding location-related image from the historically uploaded image data at the ship end based on the location keywords includes: For each location keyword, the location coordinates corresponding to the location keyword are predicted based on the coordinate prediction model corresponding to the ship end. For each image data historically uploaded from the ship, calculate the text similarity between the location keyword and the location text tag of the image data; Calculate the positional distance between the positional coordinates corresponding to the location keyword and the positional coordinates label of the image data; Calculate the weighted sum of the text similarity and the location distance to obtain the matching degree between the location keyword and the image data; The image data with the highest matching degree is determined as the location-related image corresponding to the location keyword.

[0018] As an optional implementation, in a second aspect of the invention, the specific method by which the generation module generates a standardized feedback record corresponding to the emergency response text based on the joint analysis of the location-related image and the emergency response text includes: Determine the type of emergency event corresponding to the emergency response text; Select the image recognition model corresponding to the emergency event type from the preset image recognition model library; Each location-associated image is input into the image recognition model to obtain the emergency response recognition result; the emergency response recognition result includes the emergency scene image area, the emergency response image area, and the degree of emergency response completion. The location-associated image and the corresponding emergency response identification result are filled into the standardized feedback template corresponding to the emergency event type to obtain the corresponding standardized feedback record.

[0019] As an optional implementation, in a second aspect of the invention, the specific method by which the generation module determines the emergency event type corresponding to the emergency response text includes: Determine the text location corresponding to each of the aforementioned location keywords in the emergency response text; Based on the text location, all the location keywords are clustered to obtain a keyword set; the keyword set includes multiple location keywords whose text locations are less than a preset distance threshold. The shortest text segment in the emergency response text that includes all the location keywords in the keyword set is identified as the text to be predicted; The text to be predicted is input into a trained emergency event type prediction model to obtain the emergency event type corresponding to the emergency response text; the emergency event type prediction model is trained using a training dataset that includes multiple training emergency text descriptions and corresponding emergency event type annotations.

[0020] A third aspect of this invention discloses another ship emergency feedback system based on joint image and text analysis, the system comprising: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute some or all of the steps in the ship emergency feedback method based on joint analysis of images and text disclosed in the first aspect of the present invention.

[0021] The fourth aspect of the present invention discloses a computer storage medium storing computer instructions, which, when invoked, are used to execute some or all of the steps in the ship emergency feedback method based on joint image and text analysis disclosed in the first aspect of the present invention.

[0022] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: This invention acquires emergency response text sent from the ship, identifies multiple location keywords based on a text character recognition algorithm, determines location-related images in the ship's historical image data, and generates standardized feedback records based on the joint analysis of location-related images and emergency response text. This enables intelligent location association and standardized processing of emergency response information, improves the accuracy and standardization of ship emergency feedback, and reduces the risk of feedback delays or information loss caused by manual recording. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart illustrating a ship emergency feedback method based on joint analysis of images and text disclosed in an embodiment of the present invention.

[0025] Figure 2 This is a schematic diagram of the structure of a ship emergency feedback system based on joint analysis of images and text disclosed in an embodiment of the present invention.

[0026] Figure 3 This is a schematic diagram of another ship emergency feedback system based on joint analysis of images and text disclosed in an embodiment of the present invention. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0029] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0030] This invention discloses a ship emergency feedback method and system based on joint image and text analysis. By acquiring emergency response text sent from the ship, multiple location keywords are identified using a text character recognition algorithm. Location-related images are determined from the ship's historical image data. Standardized feedback records are generated based on the joint analysis of the location-related images and the emergency response text. This enables intelligent location association and standardized processing of emergency response information, improving the accuracy and standardization of ship emergency feedback and reducing the risk of feedback delays or information loss due to manual recording. Detailed explanations follow.

[0031] Example 1 Please see Figure 1 , Figure 1 This is a flowchart illustrating a ship emergency feedback method based on joint image and text analysis disclosed in an embodiment of the present invention. Figure 1 The described ship emergency response method based on joint image and text analysis can be applied to data processing systems / data processing equipment / data processing servers (including local processing servers or cloud processing servers). Figure 1 As shown, this ship emergency response method based on joint image and text analysis may include the following operations: 101. Obtain the emergency response text sent by the ship.

[0032] Optionally, the ship end can be an edge computing gateway on an ocean-going cargo ship, a monitoring center on an offshore operation platform, or a mobile handheld terminal with satellite communication function equipped by the crew. This invention does not limit the scope of the invention.

[0033] Optionally, the emergency response text may be a description of engine room damage sent via maritime satellite short message, a fire distress message transcribed via shipborne radio, or an emergency work order for equipment failure manually entered by the crew. This invention does not impose any limitations on this.

[0034] 102. Based on text character recognition algorithms, identify multiple location keywords in emergency response texts.

[0035] 103. Based on location keywords, determine the corresponding location-related images in the historical image data uploaded by the ship.

[0036] 104. Based on the joint analysis of location-related images and emergency response texts, generate standardized feedback records corresponding to the emergency response texts.

[0037] Optionally, the standardized feedback record can be an automatically generated formatted emergency rescue plan, a maintenance guidance JSON message containing spatial coordinates, or an encrypted navigation safety event log that is triggered for reporting; this invention does not impose any limitations.

[0038] As can be seen, the above-described embodiments of the invention acquire emergency response text sent by the ship, identify multiple location keywords based on a text character recognition algorithm, determine location-related images in the ship's historical image data, and generate standardized feedback records based on the joint analysis of location-related images and emergency response text. This enables intelligent location association and standardized processing of emergency response information, improves the accuracy and standardization of ship emergency feedback, and reduces the risk of feedback delays or information loss caused by manual recording.

[0039] As an optional embodiment, the above steps, based on a text character recognition algorithm, identify multiple location keywords in the emergency response text, including: Based on the LLM model, multiple possible location representation words are generated for the ship end; Based on word segmentation algorithms, multiple text words corresponding to emergency response text are segmented. For each text word, calculate the average similarity between the text word and each possible positional character word to obtain the similarity parameter corresponding to the text word; Based on similarity parameters, multiple location keywords are selected from all text words.

[0040] Optionally, the word segmentation algorithm can be the Wirtschaft word segmentation algorithm based on Hidden Markov Model (HMM), the rule segmenter based on forward maximum matching, or the sub-word symbolization segmentation operator based on Byte Pair Encoding (BPE). This invention does not limit the algorithm.

[0041] Optionally, the above similarity calculation can be achieved by using a cosine similarity function calculated from semantic encoding vectors.

[0042] As can be seen, through the above optional embodiments, multiple possible location representation words are generated based on the LLM model, emergency response text is processed based on word segmentation algorithm to obtain text words, and the average similarity between text words and location representation words is calculated to filter location keywords. This achieves intelligent extraction of location keywords based on large language model and similarity matching, improves the targeting and efficiency of keyword recognition, and reduces the risk of keyword omission due to ambiguity in location description.

[0043] As an optional embodiment, in the above steps, generating multiple possible location representation terms corresponding to the ship's end based on the LLM model includes: Obtain the corresponding ship parameters and ship description from the ship's end; Based on preset location description prompt rules, the ship parameters and ship description are used to generate corresponding location description prompts; The location description cue words are input into the trained LLM model to obtain multiple output location description words; Calculate the repetition rate for each positional descriptor; optionally, the repetition rate is the average of the similarities between each positional descriptor and all other positional descriptors. Position descriptors with a repetition rate less than a preset repetition rate threshold are selected to obtain multiple possible position descriptors.

[0044] Optionally, the ship parameters include at least one of the following: ship model, ship personnel information, ship hardware information, and ship cargo information.

[0045] Optionally, the ship parameters can be configuration parameters of a Panamax bulk carrier, the current watchkeeping engineer and crew roster, a list of main diesel engine and auxiliary engine hardware models, or a manifest for dangerous chemical cargo; this invention does not limit the scope of these parameters.

[0046] Optionally, the location description prompt rule can be a structured text concatenation rule containing specific XML tags, or a system guidance instruction with a few-shot example of ship cabin space mapping; this invention does not limit this.

[0047] Optionally, the LLM model is trained using a training dataset that includes multiple training ship data and corresponding ship position descriptions.

[0048] As can be seen, through the above optional embodiments, by obtaining ship parameters and ship descriptions, generating prompt words based on location description prompt word rules, inputting prompt words into the trained LLM model, and filtering low-repetition location prompt words to obtain possible location representation words, accurate location description generation based on ship parameters and large language model is achieved, improving the diversity and relevance of location representation words, and reducing the risk of representation bias caused by incomplete ship descriptions.

[0049] As an optional embodiment, in the above steps, multiple location keywords are selected from all text words based on similarity parameters, including: Determine whether the similarity parameter corresponding to the text word is greater than a preset first parameter threshold, and obtain the first determination result; If the first judgment result is yes, the text word is identified as a location keyword; If the first judgment result is negative, determine whether the similarity parameter corresponding to the text word is less than the preset second parameter threshold to obtain the second judgment result; specifically, the second parameter threshold is less than the first parameter threshold. If the second judgment result is yes, the text word will not be identified as a location keyword; If the second judgment result is negative, determine whether the similarity parameter corresponding to the text word falls within the corresponding reference similarity parameter range to obtain the third judgment result; If the third judgment result is yes, the text word is identified as a location keyword; If the third judgment result is negative, the text word will not be identified as a location keyword.

[0050] As can be seen, through the above optional embodiments, the location keywords are determined by judging whether the text word similarity parameter is greater than the first threshold, less than the second threshold, or falls into the reference similarity parameter range. This achieves accurate keyword screening by combining multiple thresholds and dynamic ranges, improves the flexibility and accuracy of keyword recognition, and reduces the risk of misjudgment caused by fixed thresholds.

[0051] As an optional embodiment, the reference similarity parameter range in the above steps is calculated in the following way: Text words whose similarity parameters are greater than the first parameter threshold are identified as reference text words; Calculate the average of the similarity parameters corresponding to all reference text words to obtain the median of the interval; Calculate the standard deviation of the similarity parameters corresponding to all reference text words to obtain the interval representation value; Based on the total number of all reference text words and the pre-defined correspondence between the number and the deviation parameter, the corresponding interval deviation parameter is determined. The interval deviation value is obtained by multiplying the interval deviation parameter and the interval characterization value. The lower limit of the interval is determined by subtracting the interval deviation value from the interval median, and the upper limit of the interval is determined by adding the interval deviation value to the interval median, so as to obtain the reference similar parameter interval.

[0052] Optionally, the preset correspondence between the quantity and the deviation parameter can be calculated based on historical data, either as a mathematical polynomial model or a mapping table corresponding to the numerical interval, or it can be learned from historical data through a neural network model. Generally speaking, the more total number of all reference text words, the larger the obtained interval deviation parameter, and vice versa, so as to effectively adjust the confidence interval of the similarity parameter based on the number of texts.

[0053] As can be seen, through the above optional embodiments, the reference similarity parameter interval is obtained by calculating the median of the interval based on the reference text words and the deviation value related to the standard deviation. This realizes the determination of the similarity parameter interval of the reference confidence region calculation method, which helps to improve the flexibility and accuracy of keyword recognition and reduce the risk of misjudgment caused by fixed thresholds.

[0054] As an optional embodiment, the step above, determining the corresponding location-related image from the historically uploaded image data on the ship based on location keywords, includes: For each location keyword, the location coordinates corresponding to that location keyword are predicted based on the coordinate prediction model at the ship end. For each image data uploaded historically from the ship, calculate the text similarity between the location keyword and the location text tag of the image data; Calculate the positional distance between the positional coordinates corresponding to the location keyword and the positional coordinates label of the image data; Calculate the weighted sum of text similarity and location distance to obtain the matching degree between the keyword at that location and the image data; The image data with the highest matching degree is identified as the location-related image corresponding to the keyword at that location.

[0055] Optionally, the coordinate prediction model can be a prediction model trained on a training dataset based on multiple training location texts and corresponding location coordinate annotations. It can be a classification model or a neural network model.

[0056] Optionally, the image data can be on-site photos with Exif metadata uploaded by crew members using smart terminals during daily inspections, or key frames of video from fixed CCTV cameras that are periodically backed up to the ship's local server. This invention does not limit the scope of the data.

[0057] As can be seen, through the above optional embodiments, the location coordinates of location keywords are predicted by the ship end coordinate prediction model, the matching degree is obtained by weighted summation of text similarity and location distance with historical image data, and the image with the highest matching degree is selected as the location association image. This achieves accurate determination of location association images based on coordinate prediction and multi-dimensional matching, improves the reliability and spatial consistency of image association, and reduces the risk of association errors caused by coordinate deviation.

[0058] As an optional embodiment, the step above, generating a standardized feedback record corresponding to the emergency response text based on the joint analysis of the location-related image and the emergency response text, includes: Determine the type of emergency event corresponding to the emergency response text; Select image recognition models corresponding to the types of emergency events from the preset image recognition model library; The image associated with each location is input into the image recognition model to obtain the emergency response recognition result; optionally, the emergency response recognition result includes the emergency scene image area, the emergency response image area, and the degree of completion of emergency response; The location-associated image and the corresponding emergency response identification results are filled into the standardized feedback template corresponding to the emergency event type to obtain the corresponding standardized feedback record.

[0059] Optionally, the emergency event type can be engine room water ingress, starboard side collision damage, main engine overheating alarm, hazardous material leakage, or deck fire; this invention does not limit the event type.

[0060] Optionally, the image recognition model can be a YOLOv8-seg model pre-trained for fire smoke segmentation, a Mask R-CNN network for identifying pipe cracks, or a Faster R-CNN network for target detection of equipment damage status. This invention does not limit the model.

[0061] Optionally, the degree of completion of the emergency response can be a percentage progress value representing whether the valve is completely closed, a binary status quantity representing whether the leaking hole is blocked, or a fire extinguishing rating coefficient calculated based on the area of ​​residual open flame at the scene. This invention does not limit the specific parameters.

[0062] As can be seen, through the above optional embodiments, by determining the emergency event type corresponding to the emergency response text, the corresponding model processing location associated image is selected from the image recognition model library to obtain the emergency response recognition result, and a standardized feedback record is generated by filling the standardized feedback template. This realizes emergency image intelligent analysis and standardized record generation based on event type, improves the standardization and completeness of feedback records, and reduces the risk of record inconsistency caused by manual sorting.

[0063] As an optional embodiment, the step of determining the emergency event type corresponding to the emergency response text in the above steps includes: In the emergency response text, identify the text location corresponding to each keyword; Cluster all location keywords based on text location to obtain a keyword set; optionally, the keyword set includes multiple location keywords whose text locations are less than a preset distance threshold. The shortest text segment in the emergency response text that contains all the location keywords in the keyword set is identified as the text to be predicted; The text to be predicted is input into the trained emergency event type prediction model to obtain the emergency event type corresponding to the emergency response text.

[0064] Optionally, the emergency event type prediction model is trained using a training dataset that includes multiple training emergency text descriptions and corresponding emergency event type annotations.

[0065] Optionally, the text position can be a pair of intervals consisting of the starting character index and the ending character index of the position keyword in the emergency response text string, which is not limited in this invention.

[0066] Optionally, the keyword set can be a high-density cluster of features after the character index distance is clustered and converged by the density clustering algorithm (DBSCAN) or the one-dimensional K-Means algorithm. This invention does not limit this.

[0067] Optionally, the emergency event type prediction model can be a text classification model based on TextCNN, a sequence labeling network based on a bidirectional Transformer architecture, or a lightweight maritime language model finely tuned on a general corpus. This invention does not limit the model.

[0068] As can be seen, through the above optional embodiments, by determining the text position corresponding to the location keywords in the emergency response text, obtaining the keyword set based on text position clustering, determining the shortest text fragment containing the set as the text to be predicted, and inputting it into the trained emergency event type prediction model to obtain the emergency event type, the accurate emergency event type identification based on text position clustering and model prediction is achieved, improving the accuracy and automation of event type judgment, and reducing the risk of event misjudgment caused by keyword dispersion.

[0069] Example 2 Please see Figure 2 , Figure 2 This is a schematic diagram of a ship emergency feedback system based on joint image and text analysis, as disclosed in an embodiment of the present invention. Figure 2 The described ship emergency feedback system based on joint image and text analysis can be applied to data processing systems / data processing equipment / data processing servers (including local processing servers or cloud processing servers). For example... Figure 2 As shown, the ship emergency response system based on joint image and text analysis may include: The acquisition module 201 is used to acquire the emergency response text sent by the ship.

[0070] The recognition module 202 is used to identify multiple location keywords in emergency response text based on a text character recognition algorithm.

[0071] The determination module 203 is used to determine the corresponding location-related image in the historical image data uploaded by the ship based on location keywords.

[0072] The generation module 204 is used to generate standardized feedback records corresponding to the emergency response text based on the joint analysis of location-related images and emergency response text.

[0073] As can be seen, the above-described embodiments of the invention acquire emergency response text sent by the ship, identify multiple location keywords based on a text character recognition algorithm, determine location-related images in the ship's historical image data, and generate standardized feedback records based on the joint analysis of location-related images and emergency response text. This enables intelligent location association and standardized processing of emergency response information, improves the accuracy and standardization of ship emergency feedback, and reduces the risk of feedback delays or information loss caused by manual recording.

[0074] As an optional embodiment, the recognition module identifies multiple location keywords in emergency response text using a text character recognition algorithm, including: Based on the LLM model, multiple possible location representation words are generated for the ship end; Based on word segmentation algorithms, multiple text words corresponding to emergency response text are segmented. For each text word, calculate the average similarity between the text word and each possible positional character word to obtain the similarity parameter corresponding to the text word; Based on similarity parameters, multiple location keywords are selected from all text words.

[0075] As can be seen, through the above optional embodiments, multiple possible location representation words are generated based on the LLM model, emergency response text is processed based on word segmentation algorithm to obtain text words, and the average similarity between text words and location representation words is calculated to filter location keywords. This achieves intelligent extraction of location keywords based on large language model and similarity matching, improves the targeting and efficiency of keyword recognition, and reduces the risk of keyword omission due to ambiguity in location description.

[0076] As an optional embodiment, the specific method by which the recognition module generates multiple possible location representation words corresponding to the ship end based on the LLM model includes: Obtain the corresponding ship parameters and ship description from the ship's end; optionally, the ship parameters include at least one of the following: ship model, ship personnel information, ship hardware information, and ship cargo information; Based on preset location description prompt rules, the ship parameters and ship description are used to generate corresponding location description prompts; The location description cue words are input into the trained LLM model to obtain multiple output location description words; optionally, the LLM model is trained using a training dataset that includes multiple training ship data and corresponding ship location description annotations. Calculate the repetition rate for each positional descriptor; optionally, the repetition rate is the average of the similarities between each positional descriptor and all other positional descriptors. Position descriptors with a repetition rate less than a preset repetition rate threshold are selected to obtain multiple possible position descriptors.

[0077] As can be seen, through the above optional embodiments, by obtaining ship parameters and ship descriptions, generating prompt words based on location description prompt word rules, inputting prompt words into the trained LLM model, and filtering low-repetition location prompt words to obtain possible location representation words, accurate location description generation based on ship parameters and large language model is achieved, improving the diversity and relevance of location representation words, and reducing the risk of representation bias caused by incomplete ship descriptions.

[0078] As an optional embodiment, the specific method by which the recognition module filters multiple location keywords from all text words based on similarity parameters includes: Determine whether the similarity parameter corresponding to the text word is greater than a preset first parameter threshold, and obtain the first determination result; If the first judgment result is yes, the text word is identified as a location keyword; If the first judgment result is negative, determine whether the similarity parameter corresponding to the text word is less than the preset second parameter threshold to obtain the second judgment result; optionally, the second parameter threshold is less than the first parameter threshold. If the second judgment result is yes, the text word will not be identified as a location keyword; If the second judgment result is negative, determine whether the similarity parameter corresponding to the text word falls within the corresponding reference similarity parameter range to obtain the third judgment result; If the third judgment result is yes, the text word is identified as a location keyword; If the third judgment result is negative, the text word will not be identified as a location keyword.

[0079] As can be seen, through the above optional embodiments, the location keywords are determined by judging whether the text word similarity parameter is greater than the first threshold, less than the second threshold, or falls into the reference similarity parameter range. This achieves accurate keyword screening by combining multiple thresholds and dynamic ranges, improves the flexibility and accuracy of keyword recognition, and reduces the risk of misjudgment caused by fixed thresholds.

[0080] As an optional embodiment, the reference similarity parameter range is calculated in the following way: Text words whose similarity parameters are greater than the first parameter threshold are identified as reference text words; Calculate the average of the similarity parameters corresponding to all reference text words to obtain the median of the interval; Calculate the standard deviation of the similarity parameters corresponding to all reference text words to obtain the interval representation value; Based on the total number of all reference text words and the pre-defined correspondence between the number and the deviation parameter, the corresponding interval deviation parameter is determined. The interval deviation value is obtained by multiplying the interval deviation parameter and the interval characterization value. The lower limit of the interval is determined by subtracting the interval deviation value from the interval median, and the upper limit of the interval is determined by adding the interval deviation value to the interval median, so as to obtain the reference similar parameter interval.

[0081] As can be seen, through the above optional embodiments, the reference similarity parameter interval is obtained by calculating the median of the interval based on the reference text words and the deviation value related to the standard deviation. This realizes the determination of the similarity parameter interval of the reference confidence region calculation method, which helps to improve the flexibility and accuracy of keyword recognition and reduce the risk of misjudgment caused by fixed thresholds.

[0082] As an optional embodiment, the method by which the determining module determines the corresponding location-related image from historically uploaded image data on the ship based on location keywords includes: For each location keyword, the location coordinates corresponding to that location keyword are predicted based on the coordinate prediction model at the ship end. For each image data uploaded historically from the ship, calculate the text similarity between the location keyword and the location text tag of the image data; Calculate the positional distance between the positional coordinates corresponding to the location keyword and the positional coordinates label of the image data; Calculate the weighted sum of text similarity and location distance to obtain the matching degree between the keyword at that location and the image data; The image data with the highest matching degree is identified as the location-related image corresponding to the keyword at that location.

[0083] As can be seen, through the above optional embodiments, the location coordinates of location keywords are predicted by the ship end coordinate prediction model, the matching degree is obtained by weighted summation of text similarity and location distance with historical image data, and the image with the highest matching degree is selected as the location association image. This achieves accurate determination of location association images based on coordinate prediction and multi-dimensional matching, improves the reliability and spatial consistency of image association, and reduces the risk of association errors caused by coordinate deviation.

[0084] As an optional embodiment, the generation module generates standardized feedback records corresponding to emergency response texts based on joint analysis of location-related images and emergency response texts in the following specific ways: Determine the type of emergency event corresponding to the emergency response text; Select image recognition models corresponding to the types of emergency events from the preset image recognition model library; The image associated with each location is input into the image recognition model to obtain the emergency response recognition result; optionally, the emergency response recognition result includes the emergency scene image area, the emergency response image area, and the degree of completion of emergency response; The location-associated image and the corresponding emergency response identification results are filled into the standardized feedback template corresponding to the emergency event type to obtain the corresponding standardized feedback record.

[0085] As can be seen, through the above optional embodiments, by determining the emergency event type corresponding to the emergency response text, the corresponding model processing location associated image is selected from the image recognition model library to obtain the emergency response recognition result, and a standardized feedback record is generated by filling the standardized feedback template. This realizes emergency image intelligent analysis and standardized record generation based on event type, improves the standardization and completeness of feedback records, and reduces the risk of record inconsistency caused by manual sorting.

[0086] As an optional embodiment, the specific method by which the generation module determines the emergency event type corresponding to the emergency response text includes: In the emergency response text, identify the text location corresponding to each keyword; Cluster all location keywords based on text location to obtain a keyword set; optionally, the keyword set includes multiple location keywords whose text locations are less than a preset distance threshold. The shortest text segment in the emergency response text that contains all the location keywords in the keyword set is identified as the text to be predicted; The text to be predicted is input into the trained emergency event type prediction model to obtain the emergency event type corresponding to the emergency response text; the emergency event type prediction model is trained using a training dataset that includes multiple training emergency text descriptions and corresponding emergency event type annotations.

[0087] As can be seen, through the above optional embodiments, by determining the text position corresponding to the location keywords in the emergency response text, obtaining the keyword set based on text position clustering, determining the shortest text fragment containing the set as the text to be predicted, and inputting it into the trained emergency event type prediction model to obtain the emergency event type, the accurate emergency event type identification based on text position clustering and model prediction is achieved, improving the accuracy and automation of event type judgment, and reducing the risk of event misjudgment caused by keyword dispersion.

[0088] Example 3 Please see Figure 3 , Figure 3This is another ship emergency feedback system based on joint analysis of images and text disclosed in the embodiments of the present invention. Figure 3 The described ship emergency feedback system based on joint image and text analysis is applied in a data processing system / data processing equipment / data processing server (wherein, the server includes a local processing server or a cloud processing server). For example... Figure 3 As shown, the ship emergency response system based on joint image and text analysis may include: Memory 301 storing executable program code; Processor 302 coupled to memory 301; The processor 302 calls the executable program code stored in the memory 301 to execute the steps of the ship emergency feedback method based on image and text joint analysis described in Embodiment 1.

[0089] Example 4 This invention discloses a computer read storage medium that stores a computer program for electronic data interchange, wherein the computer program causes a computer to execute the steps of the ship emergency response method based on joint image and text analysis described in Embodiment 1.

[0090] Example 5 This invention discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to perform the steps of the ship emergency feedback method based on image and text joint analysis described in Embodiment 1.

[0091] The foregoing has described specific embodiments of this specification; other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in a different order than those shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily have to follow the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0092] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0093] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0094] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the embodiments of this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0095] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0096] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0097] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0098] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0099] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0100] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0101] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0102] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0103] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0104] Finally, it should be noted that the ship emergency feedback method and system based on joint analysis of images and text disclosed in the embodiments of the present invention are merely preferred embodiments of the present invention and are only used to illustrate the technical solutions of the present invention, not to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A ship emergency feedback method based on joint image and text analysis, characterized in that, The method includes: Obtain emergency response text sent from the ship; Based on a text character recognition algorithm, multiple location keywords in the emergency response text are identified; Based on the location keywords, the corresponding location-related images are determined from the historical image data uploaded by the ship. Based on the joint analysis of the location-related image and the emergency response text, a standardized feedback record corresponding to the emergency response text is generated.

2. The ship emergency feedback method based on joint image and text analysis according to claim 1, characterized in that, The text character recognition algorithm identifies multiple location keywords in the emergency response text, including: Based on the LLM model, multiple possible location representation words corresponding to the ship end are generated; Based on the word segmentation algorithm, the emergency response text is divided into multiple text words; For each of the text words, calculate the average similarity between the text word and each of the possible positional character words to obtain the similarity parameter corresponding to the text word; Based on the similarity parameters, multiple location keywords are selected from all the text words.

3. The ship emergency feedback method based on joint image and text analysis according to claim 2, characterized in that, The LLM model generates multiple possible location representation terms corresponding to the ship's end, including: Obtain the ship parameters and ship description corresponding to the ship terminal; the ship parameters include at least one of ship model, ship personnel information, ship hardware information, and ship cargo information; Based on preset location description prompt rules, the ship parameters and ship description are used to generate corresponding location description prompts; The location description prompts are input into the trained LLM model to obtain multiple output location descriptions; the LLM model is trained using a training dataset that includes multiple training ship data and corresponding ship location description annotations. Calculate the repetition degree corresponding to each of the location descriptors; the repetition degree is the average of the similarity between each of the location descriptors and all other location descriptors; The location descriptors with a repetition degree less than a preset repetition degree threshold are selected to obtain multiple possible location characterization words.

4. The ship emergency feedback method based on joint image and text analysis according to claim 2, characterized in that, Based on the similarity parameter, multiple location keywords are selected from all the text words, including: Determine whether the similarity parameter corresponding to the text word is greater than a preset first parameter threshold, and obtain the first determination result; If the first determination result is yes, the text word is identified as a location keyword; If the first judgment result is negative, it is determined whether the similarity parameter corresponding to the text word is less than the preset second parameter threshold to obtain the second judgment result; the second parameter threshold is less than the first parameter threshold. If the second judgment result is yes, the text word will not be identified as a location keyword; If the second judgment result is negative, determine whether the similarity parameter corresponding to the text word falls within the corresponding reference similarity parameter range to obtain the third judgment result; When the third judgment result is yes, the text word is identified as a location keyword; If the third judgment result is negative, the text word will not be identified as a location keyword.

5. The ship emergency feedback method based on joint image and text analysis according to claim 4, characterized in that, The reference similarity parameter interval is calculated in the following way: Text words whose similarity parameters are greater than the first parameter threshold are identified as reference text words; Calculate the average value of the similarity parameters corresponding to all the reference text words to obtain the median of the interval; Calculate the standard deviation of the similarity parameter corresponding to all the reference text words to obtain the interval representation value; Based on the total number of all the reference text words and the pre-defined correspondence between the number and the deviation parameter, the corresponding interval deviation parameter is determined; The interval deviation value is obtained by multiplying the interval deviation parameter and the interval characterization value. The lower limit of the interval is determined by subtracting the interval deviation value from the interval median value, and the upper limit of the interval is determined by adding the interval deviation value to the interval median value, so as to obtain the reference similarity parameter interval.

6. The ship emergency feedback method based on joint image and text analysis according to claim 1, characterized in that, The step of determining the corresponding location-related image from the historically uploaded image data on the ship based on the location keywords includes: For each location keyword, the location coordinates corresponding to the location keyword are predicted based on the coordinate prediction model corresponding to the ship end. For each image data historically uploaded from the ship, calculate the text similarity between the location keyword and the location text tag of the image data; Calculate the positional distance between the positional coordinates corresponding to the location keyword and the positional coordinates label of the image data; Calculate the weighted sum of the text similarity and the location distance to obtain the matching degree between the location keyword and the image data; The image data with the highest matching degree is determined as the location-related image corresponding to the location keyword.

7. The ship emergency feedback method based on joint image and text analysis according to claim 1, characterized in that, The step of generating a standardized feedback record corresponding to the emergency response text based on the joint analysis of the location-related image and the emergency response text includes: Determine the type of emergency event corresponding to the emergency response text; Select the image recognition model corresponding to the emergency event type from the preset image recognition model library; Each location-associated image is input into the image recognition model to obtain the emergency response recognition result; the emergency response recognition result includes the emergency scene image area, the emergency response image area, and the degree of emergency response completion. The location-associated image and the corresponding emergency response identification result are filled into the standardized feedback template corresponding to the emergency event type to obtain the corresponding standardized feedback record.

8. The ship emergency feedback method based on joint image and text analysis according to claim 7, characterized in that, Determining the emergency event type corresponding to the emergency response text includes: Determine the text location corresponding to each of the aforementioned location keywords in the emergency response text; Based on the text location, all the location keywords are clustered to obtain a keyword set; the keyword set includes multiple location keywords whose text locations are less than a preset distance threshold. The shortest text segment in the emergency response text that includes all the location keywords in the keyword set is identified as the text to be predicted; The text to be predicted is input into a trained emergency event type prediction model to obtain the emergency event type corresponding to the emergency response text; the emergency event type prediction model is trained using a training dataset that includes multiple training emergency text descriptions and corresponding emergency event type annotations.

9. A ship emergency feedback system based on joint image and text analysis, characterized in that, The system includes: The acquisition module is used to acquire emergency response texts sent from the ship. The recognition module is used to identify multiple location keywords in the emergency response text based on a text character recognition algorithm; The determination module is used to determine the corresponding location-related image in the historical image data uploaded by the ship based on the location keywords; The generation module is used to generate a standardized feedback record corresponding to the emergency response text based on the joint analysis of the location-related image and the emergency response text.

10. A ship emergency feedback system based on joint image and text analysis, characterized in that, The system includes: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the ship emergency feedback method based on joint analysis of images and text as described in any one of claims 1-8.

Images