Image recognition-based sea surface buoy anomaly identification method, device and equipment

Abstract

The application discloses a sea surface buoy abnormality identification method, device and equipment based on image recognition, which comprises the following steps: acquiring a sea surface buoy image set and obtaining a sea surface buoy identification result for each sea surface buoy image based on a multi-target detection model; acquiring a sea surface buoy distribution position area corresponding to each sea surface buoy identification result and comparing the sea surface buoy distribution position area with an initial sea surface buoy distribution position area to obtain a sea surface buoy movement identification result; if it is determined that the sea surface buoy movement identification result corresponds to the sea surface buoy having moved and the movement exceeds a preset movement distance threshold value, a corresponding sea surface buoy image is acquired to form a sea surface buoy abnormality identification image set and sea surface buoy abnormality early warning information is generated. The embodiment of the application can quickly and low-costly determine the sea surface buoy abnormality identification image set in combination with the sea surface buoy image set, the multi-target detection model and the initial sea surface buoy distribution position area, and timely generate the abnormality early warning information and send the abnormality early warning information to the user terminal for abnormality prompting.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology in artificial intelligence, and in particular to a method, apparatus and equipment for identifying anomalies in sea surface buoys based on image recognition. Background Technology

[0002] Currently, the monitoring of buoy movement in designated sea areas primarily employs satellite positioning and communication technologies. This involves installing satellite positioning modules on each buoy in the designated area for continuous monitoring. However, when using this satellite positioning method to monitor buoy movement and detect buoys exhibiting abnormal drift, a satellite positioning module needs to be installed on each buoy, increasing equipment costs. Furthermore, the computational process for determining buoy movement anomalies via satellite positioning involves the use of satellite resources, resulting in high positioning calculation costs. Summary of the Invention

[0003] This invention provides a method, apparatus, and device for identifying anomalies in sea surface buoys based on image recognition. It aims to solve the problems in the prior art where the monitoring of the movement of sea surface buoys in a designated sea area mainly adopts satellite positioning and communication technology. Each sea surface buoy needs to be equipped with a satellite positioning module, which increases the equipment cost. Moreover, the calculation process of determining the movement anomalies of buoys through satellite positioning involves the occupation of satellite resources, resulting in high positioning calculation costs.

[0004] In a first aspect, embodiments of the present invention provide a method for identifying anomalies in sea surface buoys based on image recognition, comprising:

[0005] In response to a buoy identification command, acquire the corresponding set of buoy images;

[0006] For each sea surface buoy image in the sea surface buoy image set, the sea surface buoy recognition result corresponding to the sea surface buoy image is obtained based on a pre-trained multi-target detection model;

[0007] Obtain the distribution area of ​​the sea surface buoys corresponding to the sea surface buoy identification results of each sea surface buoy image;

[0008] The distribution area of ​​the sea surface buoy in each sea surface buoy image is compared with the pre-stored initial distribution area of ​​the sea surface buoy to obtain the sea surface buoy movement recognition result corresponding to each sea surface buoy image.

[0009] If, based on the buoy movement identification results, it is determined that a buoy movement identification result corresponds to a buoy that has moved and the movement exceeds a preset movement distance threshold, then the corresponding buoy images are acquired to form a buoy anomaly identification image set.

[0010] Based on the set of images for identifying abnormal buoys on the sea surface, an early warning message for abnormal buoys on the sea surface is generated and sent to the user terminal.

[0011] Secondly, embodiments of the present invention also provide an image recognition-based device for identifying anomalies in sea surface buoys, comprising:

[0012] The sea surface buoy image set acquisition unit is used to acquire the corresponding sea surface buoy image set in response to the sea surface buoy identification command;

[0013] The sea surface buoy identification result acquisition unit is used to acquire the sea surface buoy identification result corresponding to each sea surface buoy image in the sea surface buoy image set based on a pre-trained multi-target detection model.

[0014] The buoy distribution location area acquisition unit is used to acquire the buoy distribution location area corresponding to the buoy identification result of each buoy image.

[0015] The buoy movement identification result acquisition unit is used to compare the distribution area of ​​the sea surface buoy in each sea surface buoy image with the pre-stored initial distribution area of ​​the sea surface buoy, and obtain the sea surface buoy movement identification result corresponding to each sea surface buoy image.

[0016] The sea surface buoy anomaly identification unit is used to obtain the corresponding sea surface buoy images to form a sea surface buoy anomaly identification image set if it is determined based on the sea surface buoy movement identification result that the sea surface buoy movement identification result corresponds to the sea surface buoy having moved and the movement exceeding a preset movement distance threshold.

[0017] An abnormality warning information sending unit is used to generate abnormality warning information for sea surface buoys based on the abnormality identification image set of the sea surface buoys, and send the abnormality warning information for sea surface buoys to the user terminal.

[0018] Thirdly, embodiments of the present invention also provide a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect above.

[0019] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, the computer program including program instructions that, when executed by a processor, can implement the method described in the first aspect above.

[0020] This invention provides a method, apparatus, and device for identifying anomalies in sea surface buoys based on image recognition. The method includes: in response to a sea surface buoy identification command, acquiring a corresponding set of sea surface buoy images; for each sea surface buoy image in the set, acquiring a sea surface buoy identification result corresponding to the sea surface buoy image based on a pre-trained multi-target detection model; acquiring the sea surface buoy distribution location area corresponding to the sea surface buoy identification result of each sea surface buoy image; comparing the sea surface buoy distribution location area of ​​each sea surface buoy image with a pre-stored initial distribution location area of ​​sea surface buoys to obtain sea surface buoy movement identification results corresponding to each sea surface buoy image; if, based on the sea surface buoy movement identification results, it is determined that a sea surface buoy movement identification result corresponds to a sea surface buoy that has moved and the movement exceeds a preset movement distance threshold, then acquiring the corresponding sea surface buoy images to form a set of sea surface buoy anomaly identification images; generating sea surface buoy anomaly warning information based on the set of sea surface buoy anomaly identification images, and sending the sea surface buoy anomaly warning information to a user terminal. The embodiments of the present invention can quickly and cost-effectively determine the abnormal identification image set of sea surface buoys by combining the sea surface buoy image set, multi-target detection model and the initial distribution location area of ​​sea surface buoys, and promptly generate abnormal early warning information to send to the user terminal to remind them of abnormal situations. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A schematic diagram illustrating an application scenario of the image recognition-based anomaly identification method for sea surface buoys provided in this embodiment of the invention;

[0023] Figure 2 A flowchart illustrating the image recognition-based anomaly identification method for sea surface buoys provided in an embodiment of the present invention;

[0024] Figure 3 A schematic diagram of a sub-process of the image recognition-based anomaly identification method for sea surface buoys provided in an embodiment of the present invention;

[0025] Figure 4 A schematic diagram of a sub-process of the image recognition-based anomaly identification method for sea surface buoys provided in an embodiment of the present invention;

[0026] Figure 5 A schematic diagram of a sub-process of the image recognition-based anomaly identification method for sea surface buoys provided in an embodiment of the present invention;

[0027] Figure 6A schematic block diagram of an image recognition-based anomaly detection device for sea surface buoys provided in an embodiment of the present invention;

[0028] Figure 7 A schematic block diagram of a computer device provided for an embodiment of the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0031] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0032] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0033] Please also refer to Figure 1 and Figure 2 ,in Figure 1 This is a schematic diagram illustrating a scenario of the image recognition-based anomaly identification method for sea surface buoys according to an embodiment of the present invention. Figure 2 This is a flowchart illustrating the image recognition-based anomaly identification method for sea surface buoys provided in an embodiment of the present invention. Figure 1As shown in the embodiment of the present invention, the image recognition-based method for identifying abnormal sea surface buoys is applied in server 10, which is an edge server or a cloud server. Image acquisition device 20 faces a specific seawater inlet area where multiple sea surface buoys are deployed in an array and performs corresponding image acquisition (wherein, each of the multiple sea surface buoys in the specific seawater inlet area can be connected to an adjacent sea surface buoy via a tow cable). Image acquisition device 20 is communicatively connected to server 10. Specifically, image acquisition device 20 can be a high-definition image acquisition device fixed on a high-rise building on the shore of the specific seawater inlet area, or it can be a high-definition image acquisition device mounted on an aircraft (such as a drone). Of course, in specific implementations, image acquisition device 20 is not limited to the implementation method in the above example, as long as it can achieve the function of facing a specific seawater inlet area where multiple sea surface buoys are deployed in an array and performing corresponding image acquisition. In the image acquisition device 20, image timed acquisition tasks can be pre-deployed to acquire images of sea surface buoys at fixed acquisition cycles, or the server 10 or the user terminal 30 can send sea surface buoy identification commands to the image acquisition device 20 to acquire sea surface buoy images; the user terminal 30 is also connected to the server 10 and / or the image acquisition device 20.

[0034] like Figure 2 As shown, the method includes the following steps S110-S160.

[0035] S110, in response to the buoy identification command, acquire the corresponding buoy image set.

[0036] In this embodiment, the technical solution is described with the server as the executing entity. When a user operates the server or user terminal to generate a buoy identification command in real time, the buoy identification command is first sent to the image acquisition device. Then, the image acquisition device begins image acquisition according to the buoy identification command, specifically acquiring images of a specific seawater inlet area where multiple buoys are deployed in an array, and acquiring at least one frame of buoy image, thus forming a buoy image set (which corresponds to at least one frame of buoy image). The buoy images acquired by the image acquisition device are then transmitted back to the server for buoy identification, thereby obtaining the identification result.

[0037] In practice, due to varying weather conditions (such as sunny, cloudy, rainy, and heavy rain) at the deployment location of the image acquisition device, the buoy identification command can also include weather type information and corresponding image acquisition mode switching control commands. When the server or user terminal sends the buoy identification command to the image acquisition device, the device can switch the image acquisition mode accordingly (e.g., if the weather type information is sunny, it switches to the sunny image acquisition mode; if the weather type information is heavy rain, it switches to the heavy rain image acquisition mode). This image acquisition mode switching allows the image acquisition device to acquire clearer images of the buoy under the current weather conditions, which is beneficial for more accurate image identification later.

[0038] S120. For each sea surface buoy image in the sea surface buoy image set, the sea surface buoy recognition result corresponding to the sea surface buoy image is obtained based on a pre-trained multi-target detection model.

[0039] In this embodiment, a pre-trained multi-object detection model (such as YOLO v3 or YOLO v5) is deployed on the server. This multi-object detection model can perform image recognition on each sea buoy image in the sea buoy image set, thereby obtaining a sea buoy recognition result corresponding to each sea buoy image. To facilitate understanding of the technical solution of this application, the recognition process for one sea buoy image in the sea buoy image set is used as an example. The recognition process for other sea buoy images in the sea buoy image set is also based on the above process. For example, taking sea buoy image A1 in the sea buoy image set as an example, the multi-object detection model can identify the sea buoy recognition result corresponding to sea buoy image A1.

[0040] Before inputting the buoy image into the multi-target detection model for buoy localization and detection, the buoy image can be preprocessed, such as scaling it to a preset uniform image size and normalizing it (e.g., converting the pixel value of each pixel in the buoy image to the range of 0-1). This preprocessing facilitates better feature extraction and multi-target detection in the subsequent process.

[0041] In one embodiment, such as Figure 3 As shown, step S120 includes:

[0042] S121. Based on the multi-target detection model, feature extraction is performed on the sea surface buoy image to obtain image extracted features;

[0043] S122. Based on the multi-target detection model, obtain the classification category, grid bounding box coordinates and classification category confidence for each image grid in the image feature extraction;

[0044] S123. Based on the classification category, grid bounding box coordinates and classification confidence of each image grid in the extracted image features, the results of the sea surface buoy identification are summarized and processed to obtain the identification results of the sea surface buoy.

[0045] In this embodiment, referring to the example above, the multi-target detection model can extract features from the surface buoy image A1. Specifically, feature extraction is performed through the backbone network (such as a cross-stage partial network) in the multi-target detection model (such as the YOLO v5 model) to obtain the extracted features corresponding to the surface buoy image A1. Then, the feature fusion layer in the backbone network (such as PANet, which stands for PathAggregationNetwork, an architecture used to enhance the flow of information in neural networks) is used to fuse the extracted features corresponding to the surface buoy image A1 to obtain the image extracted features.

[0046] Next, the detection head (i.e., the head detection head, which includes multiple predefined-sized anchor boxes to adaptively match the target size) in the multi-object detection model is used to extract features from the image to detect sea surface buoys. Each sea surface buoy corresponds to an image grid. The detection head can determine the classification category (such as sea surface buoy, other floating objects on the sea surface, etc.), grid bounding box coordinates (generally, the image grid is rectangular, and the grid bounding box coordinates correspond to the coordinates of the four vertices of the image grid), and classification confidence (its value ranges from 0 to 1) for each image grid.

[0047] Finally, after detecting multiple image grids in the buoy image A1 and determining the classification category, bounding box coordinates, and classification confidence score for each grid, the results can be summarized based on these parameters to obtain the buoy identification result. For example, if N1 image grids (where N1 is a positive integer) are detected in buoy image A1, they can be uniquely numbered sequentially from left to right and top to bottom, and the classification category, bounding box coordinates, and classification confidence score for each grid can be statistically analyzed. Furthermore, each grid can be treated as an entity with its bounding box coordinates, and its unique number, classification category, and classification confidence score can be used as attributes to construct a knowledge network corresponding to buoy image A1, which can then be used as the buoy identification result. It is evident that the multi-target detection model can quickly locate and identify the regions where buoys exist in buoy images, thereby obtaining the buoy identification result.

[0048] S130. Obtain the distribution area of ​​the sea surface buoys corresponding to the sea surface buoy identification results of each sea surface buoy image.

[0049] In this embodiment, after determining the buoy identification result of each buoy image in the buoy image set, referring to the above example, after the buoy detection of buoy A1 is completed and the corresponding buoy identification result is obtained, the specific distribution position of each buoy in buoy image A1 can be determined more specifically, thereby determining the corresponding buoy distribution area.

[0050] In one embodiment, such as Figure 4 As shown, step S130 includes:

[0051] S131. Based on the buoy recognition results of each buoy image, determine the grid bounding box coordinates corresponding to each buoy, and determine the buoy image distribution location information according to the grid bounding box coordinates corresponding to each buoy.

[0052] S132. Summarize and statistically analyze the distribution location information of the sea surface buoy images corresponding to each sea surface buoy to obtain the distribution location area of ​​the sea surface buoy.

[0053] In this embodiment, after the buoy identification result of each buoy image is determined by the multi-target detection model, the grid bounding box coordinates of each buoy in the buoy image can be obtained accordingly. Each grid bounding box coordinate of each buoy in the buoy image corresponds to one buoy. Once the grid bounding box coordinates of each buoy in a buoy image are known, the grid bounding box coordinates corresponding to the buoy are directly used as the buoy image distribution location information. Subsequently, after obtaining the buoy image distribution location information for all buoys included in a buoy image, the information is summarized and statistically analyzed according to the ascending order of the buoys' unique numbers to obtain the buoy distribution location area of ​​that buoy image. It should be noted that in the above processing, the location of each buoy in the image is determined based on the buoy image itself; it can also be combined with the image acquisition parameters of the image acquisition device to reconstruct the physical location of each buoy. As can be seen, the above-mentioned aggregation and processing method can accurately determine the distribution area of ​​sea surface buoys in each sea surface buoy image.

[0054] S140. Compare the distribution area of ​​the sea surface buoy in each sea surface buoy image with the pre-stored initial distribution area of ​​the sea surface buoy to obtain the sea surface buoy movement recognition result corresponding to each sea surface buoy image.

[0055] In this embodiment, after obtaining the buoy distribution area of ​​each buoy image, it can be compared with the pre-stored initial distribution area of ​​the buoys. Taking buoy image A1 as an example, after obtaining the buoy distribution area of ​​buoy image A1, the distribution position of each buoy in the image is compared with the initial distribution area of ​​the buoys (which also includes the image distribution positions corresponding to multiple buoys, i.e., the grid bounding box coordinates corresponding to the distribution area of ​​each buoy are also known). This comparison determines whether there is any abnormal movement of the buoys in buoy image A1 (such as deviating too far from the initial distribution position). Therefore, based on the above comparison process, the buoy movement recognition result corresponding to each buoy image can be quickly determined.

[0056] In one embodiment, taking the determination of buoy movement recognition results based on the distribution area of ​​buoys in a single buoy image as an example, such as... Figure 5 As shown, step S140 includes:

[0057] S141. Obtain the coordinates of the center point of each sea surface buoy distribution area in the sea surface buoy distribution location area;

[0058] S142. Obtain the initial coordinates of the center point of each sea surface buoy distribution area in the initial distribution location area of ​​the sea surface buoys;

[0059] S143. Calculate the movement distance between the coordinates of the center point of each sea surface buoy distribution area in the sea surface buoy distribution location area and the initial coordinates of the center point of the corresponding sea surface buoy distribution area in the initial sea surface buoy distribution location area, to obtain the movement identification sub-result of each sea surface buoy in the sea surface buoy distribution location area; wherein, the movement identification sub-result of each sea surface buoy in the sea surface buoy distribution location area is the image movement distance of each sea surface buoy;

[0060] S144. Multiply the movement identification sub-result of each sea surface buoy in the distribution area of ​​the sea surface buoy by the pre-stored size ratio relationship to obtain the actual movement identification sub-result of each sea surface buoy, and form the sea surface buoy movement identification result.

[0061] In this embodiment, referring to the example above, after determining the distribution area of ​​the buoys in the buoy image A1, the coordinates of the center point of each buoy distribution area in the buoy distribution area can be obtained in ascending order of the unique numbers. For example, the buoy at the top left of the buoy image A1 is the buoy with the unique number 1. The coordinates of the center point of its buoy distribution area can be determined based on the grid boundary frame coordinates of the buoy with the unique number 1 (e.g., summing the coordinates of the four vertices in the grid boundary frame coordinates of the buoy with the unique number 1 and then dividing by 4 will give the coordinates of the center point of its buoy distribution area). Similarly, the initial coordinates of the center point of the buoy distribution area corresponding to the buoy with the unique number 1 in the initial distribution area of ​​the buoy can be obtained (for example, in the standard buoy image corresponding to the initial distribution area of ​​the buoy, the sum of the coordinates of the four vertices in the grid boundary coordinates of the buoy with the unique number 1 and then dividing by 4 can be used to obtain the initial coordinates of the center point of the buoy distribution area).

[0062] Next, the coordinates of the center point of the distribution area of ​​the uniquely numbered buoy (1) are compared with the initial coordinates of the center point of the distribution area of ​​the uniquely numbered buoy (1) in the initial distribution area of ​​the buoy. Alternatively, the magnitude of the vector obtained by subtracting the initial coordinates from the center point coordinates can be directly calculated and used as the motion identification sub-result of the uniquely numbered buoy (i.e., the image movement distance of the uniquely numbered buoy). Finally, to restore this motion identification sub-result to the actual movement distance of the buoy in the physical world, the size ratio corresponding to the current image acquisition mode of the image acquisition device can be obtained. The motion identification sub-result of the uniquely numbered buoy (1) is then used to obtain the actual motion identification sub-result of the uniquely numbered buoy (1) based on this size ratio. After obtaining the actual motion identification sub-result of each buoy in buoy image A1 in the same way, the union of multiple actual motion identification sub-results can be used as the buoy motion identification result of buoy image A1. The process of identifying the movement of buoys in other buoy images within the buoy image set can be referenced from the process of obtaining the movement identification result of buoy image A1. It is evident that the above traversal method can quickly and accurately determine the movement identification result of each buoy image in the buoy image set.

[0063] In one embodiment, the method further includes the following after step S140:

[0064] Based on the aforementioned buoy movement identification results, obtain the actual movement identification sub-result for each buoy;

[0065] If it is determined that the actual movement identification result of a sea surface buoy is greater than the preset actual movement distance threshold, then the movement identification result of the corresponding sea surface buoy is determined to be that the sea surface buoy has moved and the movement exceeds the preset movement distance threshold.

[0066] If it is determined that the actual movement identification result of a sea surface buoy is less than or equal to the preset actual movement distance threshold, then the movement identification result of the corresponding sea surface buoy is determined to mean that the sea surface buoy has moved and the movement has not exceeded the preset movement distance threshold.

[0067] In this embodiment, referring to the example above, after determining the actual movement identification result of each buoy in the buoy image A1, the actual movement identification result of each buoy can be compared with a preset actual movement distance threshold. If the actual movement identification result of a buoy is greater than the preset actual movement distance threshold, it indicates that the buoy has moved abnormally (the cause may be that the towing cable connecting it to other buoys has broken; since the slender structure of the towing cable is difficult to detect and locate through image recognition, the towing cable between buoys can be indirectly determined by identifying whether the spacing between buoys has changed abnormally). In this case, it is determined that the movement identification result of the buoy corresponds to the buoy having moved and the movement exceeding the preset movement distance threshold. If the actual movement identification result of a buoy is less than or equal to the preset actual movement distance threshold, it indicates that the buoy may have moved within the allowable range. In this case, it is determined that the movement identification result of the buoy corresponds to the buoy having moved and the movement not exceeding the preset movement distance threshold. As can be seen, the above comparison method can be used to quickly determine the abnormal movement of each buoy in the buoy image.

[0068] S150. If, based on the buoy movement identification results, it is determined that a buoy movement identification result corresponds to a buoy that has moved and the movement exceeds a preset movement distance threshold, then the corresponding buoy images are acquired to form a buoy anomaly identification image set.

[0069] In this embodiment, if the buoy movement identification result determines that a buoy has moved beyond a preset movement distance threshold, it indicates that the buoy is moving abnormally. At this point, the corresponding buoy image is acquired. After filtering out all buoy images that meet the above conditions, these buoy images can form a buoy anomaly identification image set.

[0070] In one embodiment, the method further includes the following after step S150:

[0071] If it is determined that the similarity between the buoy movement recognition results corresponding to any two buoy images in the buoy anomaly recognition image set exceeds a preset similarity threshold, then one buoy anomaly recognition image is randomly selected from the buoy anomaly recognition image set and retained to update the buoy anomaly recognition image set.

[0072] In this embodiment, if the buoy image set consists of multiple buoy images acquired by the image acquisition device within a preset acquisition duration (e.g., 1 minute, 2 minutes, etc.) and a preset acquisition cycle (e.g., 5 seconds, 10 seconds, etc.) after receiving a buoy identification command, the buoy movement identification result corresponding to each buoy image in the buoy image set can be comprehensively analyzed. Initially, all candidate buoy images in the buoy image set that satisfy the buoy movement identification result and indicate that the buoy has moved beyond a preset movement distance threshold can be selected, thereby obtaining the buoy anomaly identification image set. Subsequently, if the similarity between the buoy movement identification results of any two buoy images in the buoy anomaly identification image set exceeds a preset similarity threshold (such as 80% or 90% set according to actual needs), it indicates that the final buoy movement anomaly identification result is highly reliable. In this case, any buoy image can be randomly selected from the buoy anomaly identification image set as the target buoy image, and the buoy movement identification result of that target buoy image can be used as the buoy movement identification result corresponding to the buoy anomaly identification image set. Therefore, based on the above method, after comprehensively determining the presence of anomalies in the buoys through multiple frames of buoy images in the buoy image set, and then selecting one frame of buoy image to form the buoy anomaly identification image set, the data volume of the buoy anomaly identification image set is effectively reduced.

[0073] S160. Generate a sea surface buoy anomaly warning information based on the sea surface buoy anomaly identification image set, and send the sea surface buoy anomaly warning information to the user terminal.

[0074] In this embodiment, after determining the buoy anomaly identification image set in the server by combining the buoy image set, multi-target detection model, and the initial distribution area of ​​the buoys, the unique number of the buoy that has moved abnormally and the corresponding actual movement identification sub-result (i.e., actual movement distance) can be determined from the buoy anomaly identification image set. This information is then filled into a preset warning information template to generate a buoy anomaly warning information. Subsequently, the buoy anomaly warning information is sent to the user terminal, promptly alerting the user to the abnormal movement of the buoy and enabling them to take timely measures to restore the abnormally moving buoy to its original position.

[0075] In one embodiment, the method further includes the following after step S160:

[0076] If, based on the multi-target detection model, it is determined that the sea surface buoy anomaly identification image in the sea surface buoy anomaly identification image set has a grid bounding box corresponding to a sea surface floating object classification type, and the ratio of the total area of ​​the grid bounding box of the sea surface floating object classification type to the image area of ​​the sea surface buoy anomaly identification image exceeds a preset area ratio threshold, then a sea surface floating object anomaly warning information is generated and sent to the user terminal.

[0077] In this embodiment, after the server obtains a set of buoy anomaly identification images in the current round of detection (each detection of a buoy identification command and subsequent complete detection of buoy movement anomalies in the server can be considered as one round of detection), a buoy anomaly identification image can be selected from it, and then the multi-object detection model can be used to determine whether the buoy anomaly identification image has a grid bounding box corresponding to the classification type of floating objects on the sea surface (the specific process can refer to the process of determining the buoy in the buoy image based on the multi-object detection model). If it is determined that the abnormal identification image of the sea surface buoy has a grid bounding box corresponding to a sea surface floating object classification type (the sea surface floating object corresponding to the classification type is different from the sea surface buoy and can be marine debris, etc.), and the ratio of the total area of ​​the grid bounding box of the sea surface floating object classification type to the image area of ​​the abnormal identification image of the sea surface buoy exceeds a preset area ratio threshold, it indicates that a large amount of sea surface floating object has been stationed in a specific seawater inlet area. The abnormal warning information of sea surface floating object is generated in accordance with the method of generating abnormal warning information of sea surface buoy and sent to the user terminal to prompt the relevant personnel to take measures to clean up the sea surface floating object.

[0078] As can be seen, the implementation of this method can quickly and cost-effectively determine the anomaly identification image set of sea surface buoys by combining the sea surface buoy image set, multi-target detection model and the initial distribution area of ​​sea surface buoys, and promptly generate anomaly warning information to send to the user terminal to remind them of abnormal situations.

[0079] Figure 6 This is a schematic block diagram of an image recognition-based anomaly detection device for sea surface buoys provided in an embodiment of the present invention. Figure 6 As shown, corresponding to the above-described image recognition-based method for identifying anomalies in sea surface buoys, the present invention also provides an image recognition-based device 100 for identifying anomalies in sea surface buoys. This image recognition-based device 100 includes a unit for performing the above-described image recognition-based method for identifying anomalies in sea surface buoys. Please refer to... Figure 6 The image recognition-based sea surface buoy anomaly identification device 100 includes: a sea surface buoy image set acquisition unit 110, a sea surface buoy identification result acquisition unit 120, a buoy distribution location area acquisition unit 130, a buoy movement identification result acquisition unit 140, a sea surface buoy anomaly identification unit 150, and an anomaly warning information sending unit 160.

[0080] The sea surface buoy image set acquisition unit 110 is used to acquire the corresponding sea surface buoy image set in response to the sea surface buoy identification command.

[0081] In this embodiment, the technical solution is described with the server as the executing entity. When a user operates the server or user terminal to generate a buoy identification command in real time, the buoy identification command is first sent to the image acquisition device. Then, the image acquisition device begins image acquisition according to the buoy identification command, specifically acquiring images of a specific seawater inlet area where multiple buoys are deployed in an array, and acquiring at least one frame of buoy image, thus forming a buoy image set (which corresponds to at least one frame of buoy image). The buoy images acquired by the image acquisition device are then transmitted back to the server for buoy identification, thereby obtaining the identification result.

[0082] In practice, due to varying weather conditions (such as sunny, cloudy, rainy, and heavy rain) at the deployment location of the image acquisition device, the buoy identification command can also include weather type information and corresponding image acquisition mode switching control commands. When the server or user terminal sends the buoy identification command to the image acquisition device, the device can switch the image acquisition mode accordingly (e.g., if the weather type information is sunny, it switches to the sunny image acquisition mode; if the weather type information is heavy rain, it switches to the heavy rain image acquisition mode). This image acquisition mode switching allows the image acquisition device to acquire clearer images of the buoy under the current weather conditions, which is beneficial for more accurate image identification later.

[0083] The sea surface buoy identification result acquisition unit 120 is used to acquire the sea surface buoy identification result corresponding to each sea surface buoy image in the sea surface buoy image set based on a pre-trained multi-target detection model.

[0084] In this embodiment, a pre-trained multi-object detection model (such as YOLO v3 or YOLO v5) is deployed on the server. This multi-object detection model can perform image recognition on each sea buoy image in the sea buoy image set, thereby obtaining a sea buoy recognition result corresponding to each sea buoy image. To facilitate understanding of the technical solution of this application, the recognition process for one sea buoy image in the sea buoy image set is used as an example. The recognition process for other sea buoy images in the sea buoy image set is also based on the above process. For example, taking sea buoy image A1 in the sea buoy image set as an example, the multi-object detection model can identify the sea buoy recognition result corresponding to sea buoy image A1.

[0085] Before inputting the buoy image into the multi-target detection model for buoy localization and detection, the buoy image can be preprocessed, such as scaling it to a preset uniform image size and normalizing it (e.g., converting the pixel value of each pixel in the buoy image to the range of 0-1). This preprocessing facilitates better feature extraction and multi-target detection in the subsequent process.

[0086] In one embodiment, the buoy identification result acquisition unit 120 is specifically used for:

[0087] Based on the multi-target detection model, feature extraction is performed on the sea surface buoy image to obtain image extracted features;

[0088] Based on the multi-object detection model, the classification category, grid bounding box coordinates, and classification category confidence are obtained for each image grid in the image feature extraction.

[0089] The classification category, grid bounding box coordinates, and classification confidence scores of each image grid in the extracted image features are summarized to obtain the buoy identification result.

[0090] In this embodiment, referring to the example above, the multi-target detection model can extract features from the surface buoy image A1. Specifically, feature extraction is performed through the backbone network (such as a cross-stage partial network) in the multi-target detection model (such as the YOLO v5 model) to obtain the extracted features corresponding to the surface buoy image A1. Then, the feature fusion layer in the backbone network (such as PANet, which stands for PathAggregationNetwork, an architecture used to enhance the flow of information in neural networks) is used to fuse the extracted features corresponding to the surface buoy image A1 to obtain the image extracted features.

[0091] Next, the detection head (i.e., the head detection head, which includes multiple predefined-sized anchor boxes to adaptively match the target size) in the multi-object detection model is used to extract features from the image to detect sea surface buoys. Each sea surface buoy corresponds to an image grid. The detection head can determine the classification category (such as sea surface buoy, other floating objects on the sea surface, etc.), grid bounding box coordinates (generally, the image grid is rectangular, and the grid bounding box coordinates correspond to the coordinates of the four vertices of the image grid), and classification confidence (its value ranges from 0 to 1) for each image grid.

[0092] Finally, after detecting multiple image grids in the buoy image A1 and determining the classification category, bounding box coordinates, and classification confidence score for each grid, the results can be summarized based on these parameters to obtain the buoy identification result. For example, if N1 image grids (where N1 is a positive integer) are detected in buoy image A1, they can be uniquely numbered sequentially from left to right and top to bottom, and the classification category, bounding box coordinates, and classification confidence score for each grid can be statistically analyzed. Furthermore, each grid can be treated as an entity with its bounding box coordinates, and its unique number, classification category, and classification confidence score can be used as attributes to construct a knowledge network corresponding to buoy image A1, which can then be used as the buoy identification result. It is evident that the multi-target detection model can quickly locate and identify the regions where buoys exist in buoy images, thereby obtaining the buoy identification result.

[0093] The buoy distribution location area acquisition unit 130 is used to acquire the buoy distribution location area corresponding to the buoy identification result of each buoy image.

[0094] In this embodiment, after determining the buoy identification result of each buoy image in the buoy image set, referring to the above example, after the buoy detection of buoy A1 is completed and the corresponding buoy identification result is obtained, the specific distribution position of each buoy in buoy image A1 can be determined more specifically, thereby determining the corresponding buoy distribution area.

[0095] In one embodiment, the buoy distribution location area acquisition unit 130 is specifically used for:

[0096] Based on the buoy recognition results of each buoy image, determine the coordinates of the grid bounding box corresponding to each buoy, and determine the distribution location information of the buoy images based on the coordinates of the grid bounding box corresponding to each buoy.

[0097] The distribution location information of each sea surface buoy image is summarized and statistically analyzed to obtain the distribution location area of ​​the sea surface buoy.

[0098] In this embodiment, after the buoy identification result of each buoy image is determined by the multi-target detection model, the grid bounding box coordinates of each buoy in the buoy image can be obtained accordingly. Each grid bounding box coordinate of each buoy in the buoy image corresponds to one buoy. Once the grid bounding box coordinates of each buoy in a buoy image are known, the grid bounding box coordinates corresponding to the buoy are directly used as the buoy image distribution location information. Subsequently, after obtaining the buoy image distribution location information for all buoys included in a buoy image, the information is summarized and statistically analyzed according to the ascending order of the buoys' unique numbers to obtain the buoy distribution location area of ​​that buoy image. It should be noted that in the above processing, the location of each buoy in the image is determined based on the buoy image itself; it can also be combined with the image acquisition parameters of the image acquisition device to reconstruct the physical location of each buoy. As can be seen, the above-mentioned aggregation and processing method can accurately determine the distribution area of ​​sea surface buoys in each sea surface buoy image.

[0099] The buoy movement recognition result acquisition unit 140 is used to compare the buoy distribution area of ​​each buoy image with the pre-stored initial distribution area of ​​the buoys to obtain the buoy movement recognition result corresponding to each buoy image.

[0100] In this embodiment, after obtaining the buoy distribution area of ​​each buoy image, it can be compared with the pre-stored initial distribution area of ​​the buoys. Taking buoy image A1 as an example, after obtaining the buoy distribution area of ​​buoy image A1, the distribution position of each buoy in the image is compared with the initial distribution area of ​​the buoys (which also includes the image distribution positions corresponding to multiple buoys, i.e., the grid bounding box coordinates corresponding to the distribution area of ​​each buoy are also known). This comparison determines whether there is any abnormal movement of the buoys in buoy image A1 (such as deviating too far from the initial distribution position). Therefore, based on the above comparison process, the buoy movement recognition result corresponding to each buoy image can be quickly determined.

[0101] In one embodiment, taking the determination of buoy movement recognition results based on the distribution area of ​​buoys in a buoy image as an example, the buoy movement recognition result acquisition unit 140 is specifically used for:

[0102] Obtain the coordinates of the center point of each sea surface buoy distribution area within the aforementioned sea surface buoy distribution location area;

[0103] Obtain the initial coordinates of the center point of each sea surface buoy distribution area within the initial distribution location region of the sea surface buoys;

[0104] The movement distance of each buoy in the distribution area is calculated by comparing the coordinates of the center point of each buoy distribution area with the initial coordinates of the center point of the corresponding buoy distribution area in the initial distribution area of ​​the buoys, thus obtaining the movement identification sub-result of each buoy in the distribution area of ​​the buoys; wherein, the movement identification sub-result of each buoy in the distribution area of ​​the buoys is the image movement distance of each buoy.

[0105] Multiply the movement identification sub-result of each sea surface buoy in the distribution area of ​​the sea surface buoy by the pre-stored size ratio to obtain the actual movement identification sub-result of each sea surface buoy, and form the sea surface buoy movement identification result.

[0106] In this embodiment, referring to the example above, after determining the distribution area of ​​the buoys in the buoy image A1, the coordinates of the center point of each buoy distribution area in the buoy distribution area can be obtained in ascending order of the unique numbers. For example, the buoy at the top left of the buoy image A1 is the buoy with the unique number 1. The coordinates of the center point of its buoy distribution area can be determined based on the grid boundary frame coordinates of the buoy with the unique number 1 (e.g., summing the coordinates of the four vertices in the grid boundary frame coordinates of the buoy with the unique number 1 and then dividing by 4 will give the coordinates of the center point of its buoy distribution area). Similarly, the initial coordinates of the center point of the buoy distribution area corresponding to the buoy with the unique number 1 in the initial distribution area of ​​the buoy can be obtained (for example, in the standard buoy image corresponding to the initial distribution area of ​​the buoy, the sum of the coordinates of the four vertices in the grid boundary coordinates of the buoy with the unique number 1 and then dividing by 4 can be used to obtain the initial coordinates of the center point of the buoy distribution area).

[0107] Next, the coordinates of the center point of the distribution area of ​​the uniquely numbered buoy (1) are compared with the initial coordinates of the center point of the distribution area of ​​the uniquely numbered buoy (1) in the initial distribution area of ​​the buoy. Alternatively, the magnitude of the vector obtained by subtracting the initial coordinates from the center point coordinates can be directly calculated and used as the motion identification sub-result of the uniquely numbered buoy (i.e., the image movement distance of the uniquely numbered buoy). Finally, to restore this motion identification sub-result to the actual movement distance of the buoy in the physical world, the size ratio corresponding to the current image acquisition mode of the image acquisition device can be obtained. The motion identification sub-result of the uniquely numbered buoy (1) is then used to obtain the actual motion identification sub-result of the uniquely numbered buoy (1) based on this size ratio. After obtaining the actual motion identification sub-result of each buoy in buoy image A1 in the same way, the union of multiple actual motion identification sub-results can be used as the buoy motion identification result of buoy image A1. The process of identifying the movement of buoys in other buoy images within the buoy image set can be referenced from the process of obtaining the movement identification result of buoy image A1. It is evident that the above traversal method can quickly and accurately determine the movement identification result of each buoy image in the buoy image set.

[0108] In one embodiment, the image recognition-based buoy anomaly detection device 100 further includes:

[0109] The actual movement identification sub-result determination unit is used to obtain the actual movement identification sub-result of each sea surface buoy based on the sea surface buoy movement identification result;

[0110] The first determination unit is used to determine that if the actual movement identification result of a sea surface buoy is greater than a preset actual movement distance threshold, the movement identification result of the corresponding sea surface buoy corresponds to the sea surface buoy having moved and the movement exceeding the preset movement distance threshold.

[0111] The second determination unit is used to determine that if the actual movement identification result of a sea surface buoy is less than or equal to a preset actual movement distance threshold, the movement identification result of the corresponding sea surface buoy corresponds to the sea surface buoy having moved and the movement not exceeding the preset movement distance threshold.

[0112] In this embodiment, referring to the example above, after determining the actual movement identification result of each buoy in the buoy image A1, the actual movement identification result of each buoy can be compared with a preset actual movement distance threshold. If the actual movement identification result of a buoy is greater than the preset actual movement distance threshold, it indicates that the buoy has moved abnormally (the cause may be that the towing cable connecting it to other buoys has broken; since the slender structure of the towing cable is difficult to detect and locate through image recognition, the towing cable between buoys can be indirectly determined by identifying whether the spacing between buoys has changed abnormally). In this case, it is determined that the movement identification result of the buoy corresponds to the buoy having moved and the movement exceeding the preset movement distance threshold. If the actual movement identification result of a buoy is less than or equal to the preset actual movement distance threshold, it indicates that the buoy may have moved within the allowable range. In this case, it is determined that the movement identification result of the buoy corresponds to the buoy having moved and the movement not exceeding the preset movement distance threshold. As can be seen, the above comparison method can be used to quickly determine the abnormal movement of each buoy in the buoy image.

[0113] The sea surface buoy anomaly identification unit 150 is used to acquire the corresponding sea surface buoy images to form a sea surface buoy anomaly identification image set if it is determined based on the sea surface buoy movement identification result that the sea surface buoy movement identification result corresponds to the sea surface buoy having moved and the movement exceeding a preset movement distance threshold.

[0114] In this embodiment, if the buoy movement identification result determines that a buoy has moved beyond a preset movement distance threshold, it indicates that the buoy is moving abnormally. At this point, the corresponding buoy image is acquired. After filtering out all buoy images that meet the above conditions, these buoy images can form a buoy anomaly identification image set.

[0115] In one embodiment, the image recognition-based buoy anomaly detection device 100 further includes:

[0116] The image set update unit is used to select any one of the sea surface buoy anomaly recognition images from the sea surface buoy anomaly recognition image set and retain it to update the sea surface buoy anomaly recognition image set if it is determined that the similarity between the sea surface buoy movement recognition results corresponding to any two sea surface buoy images in the sea surface buoy anomaly recognition image set exceeds a preset similarity threshold.

[0117] In this embodiment, if the buoy image set consists of multiple buoy images acquired by the image acquisition device within a preset acquisition duration (e.g., 1 minute, 2 minutes, etc.) and a preset acquisition cycle (e.g., 5 seconds, 10 seconds, etc.) after receiving a buoy identification command, the buoy movement identification result corresponding to each buoy image in the buoy image set can be comprehensively analyzed. Initially, all candidate buoy images in the buoy image set that satisfy the buoy movement identification result and indicate that the buoy has moved beyond a preset movement distance threshold can be selected, thereby obtaining the buoy anomaly identification image set. Subsequently, if the similarity between the buoy movement identification results of any two buoy images in the buoy anomaly identification image set exceeds a preset similarity threshold (such as 80% or 90% set according to actual needs), it indicates that the final buoy movement anomaly identification result is highly reliable. In this case, any buoy image can be randomly selected from the buoy anomaly identification image set as the target buoy image, and the buoy movement identification result of that target buoy image can be used as the buoy movement identification result corresponding to the buoy anomaly identification image set. Therefore, based on the above method, after comprehensively determining the presence of anomalies in the buoys through multiple frames of buoy images in the buoy image set, and then selecting one frame of buoy image to form the buoy anomaly identification image set, the data volume of the buoy anomaly identification image set is effectively reduced.

[0118] The abnormal warning information sending unit 160 is used to generate abnormal warning information for sea surface buoys based on the abnormal identification image set of sea surface buoys, and send the abnormal warning information for sea surface buoys to the user terminal.

[0119] In this embodiment, after determining the buoy anomaly identification image set in the server by combining the buoy image set, multi-target detection model, and the initial distribution area of ​​the buoys, the unique number of the buoy that has moved abnormally and the corresponding actual movement identification sub-result (i.e., actual movement distance) can be determined from the buoy anomaly identification image set. This information is then filled into a preset warning information template to generate a buoy anomaly warning information. Subsequently, the buoy anomaly warning information is sent to the user terminal, promptly alerting the user to the abnormal movement of the buoy and enabling them to take timely measures to restore the abnormally moving buoy to its original position.

[0120] In one embodiment, the image recognition-based buoy anomaly detection device 100 further includes:

[0121] The sea surface floating object anomaly early warning information generation unit is used to generate sea surface floating object anomaly early warning information and send it to the user terminal if, based on the multi-target detection model, it is determined that there is a sea surface floating object classification type corresponding to the grid bounding box in the sea surface floating object anomaly recognition image set, and the ratio of the total area of ​​the grid bounding box of the sea surface floating object classification type to the image area of ​​the sea surface floating object anomaly recognition image exceeds a preset area ratio threshold.

[0122] In this embodiment, after the server obtains a set of buoy anomaly identification images in the current round of detection (each detection of a buoy identification command and subsequent complete detection of buoy movement anomalies in the server can be considered as one round of detection), a buoy anomaly identification image can be selected from it, and then the multi-object detection model can be used to determine whether the buoy anomaly identification image has a grid bounding box corresponding to the classification type of floating objects on the sea surface (the specific process can refer to the process of determining the buoy in the buoy image based on the multi-object detection model). If it is determined that the abnormal identification image of the sea surface buoy has a grid bounding box corresponding to a sea surface floating object classification type (the sea surface floating object corresponding to the classification type is different from the sea surface buoy and can be marine debris, etc.), and the ratio of the total area of ​​the grid bounding box of the sea surface floating object classification type to the image area of ​​the abnormal identification image of the sea surface buoy exceeds a preset area ratio threshold, it indicates that a large amount of sea surface floating object has been stationed in a specific seawater inlet area. The abnormal warning information of sea surface floating object is generated in accordance with the method of generating abnormal warning information of sea surface buoy and sent to the user terminal to prompt the relevant personnel to take measures to clean up the sea surface floating object.

[0123] As can be seen, the implementation of this device can quickly and cost-effectively determine the abnormal buoy image set by combining the buoy image set, multi-target detection model and the initial distribution area of ​​the buoys, and promptly generate abnormal early warning information to send to the user terminal to remind them of abnormal situations.

[0124] The aforementioned image recognition-based anomaly detection device for sea surface buoys can be implemented as a computer program, which can, for example... Figure 7 It runs on the computer device shown.

[0125] Please see Figure 7 , Figure 7 This is a schematic block diagram of a computer device provided in an embodiment of the present invention. The computer device integrates any of the image recognition-based anomaly detection devices for sea surface buoys provided in the embodiments of the present invention.

[0126] See Figure 7The computer device 400 includes a processor 402, a memory, and a network interface 405 connected via a system bus 401. The memory may include a storage medium 403 and internal memory 404.

[0127] The storage medium 403 may store an operating system 4031 and a computer program 4032. The computer program 4032 includes program instructions that, when executed, cause the processor 402 to perform an image recognition-based method for identifying anomalies in sea surface buoys.

[0128] The processor 402 provides computing and control capabilities to support the operation of the entire computer device.

[0129] The internal memory 404 provides an environment for the computer program 4032 in the storage medium 403 to run. When the computer program 4032 is executed by the processor 402, the processor 402 can execute the above-mentioned image recognition-based method for identifying anomalies in sea surface buoys.

[0130] This network interface 405 is used for network communication with other devices. Those skilled in the art will understand that... Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the computer device to which the present invention is applied. A specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0131] The processor 402 is used to run a computer program 4032 stored in a memory to implement the image recognition-based method for identifying anomalies in sea surface buoys as described above.

[0132] It should be understood that, in this embodiment of the invention, the processor 402 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0133] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0134] Therefore, the present invention also provides a computer-readable storage medium. This computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When executed by a processor, the program instructions cause the processor to perform the image recognition-based method for identifying anomalies in sea surface buoys as described above.

[0135] The storage medium can be any computer-readable storage medium that can store program code, such as a USB flash drive, external hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0136] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0137] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0138] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the device of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0139] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0140] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for identifying anomalies in sea surface buoys based on image recognition, characterized in that, include: In response to a buoy identification command, acquire the corresponding set of buoy images; The set of sea surface buoy images is obtained by the image acquisition device facing the seawater inlet area with multiple sea surface buoys; For each buoy image in the aforementioned buoy image set, a buoy identification result corresponding to the buoy image is obtained based on a pre-trained multi-object detection model. The process of obtaining the buoy identification result based on the pre-trained multi-object detection model includes: determining the classification category, grid bounding box coordinates, and classification confidence score of each buoy image's image grid using the detection head in the multi-object detection model; each buoy corresponds to one image grid. The process involves obtaining the distribution area of ​​the sea surface buoys corresponding to the buoy identification results of each sea surface buoy image. This process includes: after obtaining the distribution location information of all sea surface buoys included in a sea surface buoy image, summarizing and statistically analyzing the information according to the ascending order of the unique numbers of the sea surface buoys to obtain the distribution area of ​​the sea surface buoys in that sea surface buoy image. The distribution area of ​​the sea surface buoy in each sea surface buoy image is compared with the pre-stored initial distribution area of ​​the sea surface buoy to obtain the sea surface buoy movement recognition result corresponding to each sea surface buoy image. If, based on the buoy movement identification results, it is determined that a buoy movement identification result corresponds to a buoy that has moved and the movement exceeds a preset movement distance threshold, then the corresponding buoy images are acquired to form a buoy anomaly identification image set. Based on the set of images for identifying abnormal buoys on the sea surface, an early warning message for abnormal buoys on the sea surface is generated and sent to the user terminal. The step of comparing the buoy distribution area of ​​each buoy image with the pre-stored initial distribution area of ​​buoys to obtain the buoy movement recognition result corresponding to each buoy image includes: Obtain the coordinates of the center point of each sea surface buoy distribution area within the aforementioned sea surface buoy distribution location area; Obtain the initial coordinates of the center point of each sea surface buoy distribution area within the initial distribution location region of the sea surface buoys; The movement distance of each buoy in the distribution area is calculated by comparing the coordinates of the center point of each buoy distribution area with the initial coordinates of the center point of the corresponding buoy distribution area in the initial distribution area of ​​the buoys, thus obtaining the movement identification sub-result of each buoy in the distribution area of ​​the buoys; wherein, the movement identification sub-result of each buoy in the distribution area of ​​the buoys is the image movement distance of each buoy. Multiply the movement identification sub-result of each sea surface buoy in the distribution area of ​​the sea surface buoy by the pre-stored size ratio to obtain the actual movement identification sub-result of each sea surface buoy, and combine them to form the sea surface buoy movement identification result; After the step of comparing the buoy distribution area of ​​each buoy image with the pre-stored initial distribution area of ​​buoys to obtain the buoy movement recognition result corresponding to each buoy image, the method further includes: Based on the aforementioned buoy movement identification results, obtain the actual movement identification sub-result for each buoy; If it is determined that the actual movement identification result of a sea surface buoy is greater than the preset actual movement distance threshold, then it is determined that the movement identification result of the corresponding sea surface buoy corresponds to the sea surface buoy having moved and the movement exceeding the preset movement distance threshold, and it is determined that the towing cable connecting the corresponding sea surface buoy to other sea surface buoys has broken. If it is determined that the actual movement identification result of a sea surface buoy is less than or equal to the preset actual movement distance threshold, then the movement identification result of the corresponding sea surface buoy is determined to be that the sea surface buoy has moved and the movement has not exceeded the preset movement distance threshold. After the steps of generating sea surface buoy anomaly warning information based on the sea surface buoy anomaly identification image set and sending the sea surface buoy anomaly warning information to the user terminal, the method further includes: If, based on the multi-target detection model, it is determined that the sea surface buoy anomaly identification image in the sea surface buoy anomaly identification image set has a grid bounding box corresponding to a sea surface floating object classification type, and the ratio of the total area of ​​the grid bounding box of the sea surface floating object classification type to the image area of ​​the sea surface buoy anomaly identification image exceeds a preset area ratio threshold, then a sea surface floating object anomaly warning information is generated and sent to the user terminal.

2. The method according to claim 1, characterized in that, The pre-trained multi-target detection model acquires the buoy identification result corresponding to the buoy image, including: Based on the multi-target detection model, feature extraction is performed on the sea surface buoy image to obtain image extracted features; Based on the multi-object detection model, the classification category, grid bounding box coordinates, and classification category confidence are obtained for each image grid in the image feature extraction. The classification category, grid bounding box coordinates, and classification confidence scores of each image grid in the extracted image features are summarized to obtain the buoy identification result.

3. The method according to claim 1, characterized in that, The area of ​​the distribution location of the sea surface buoys corresponding to the sea surface buoy identification result obtained from each sea surface buoy image includes: Based on the buoy identification results of each buoy image, the coordinates of the grid bounding box corresponding to each buoy are determined, and the distribution location information of the buoy images is determined according to the coordinates of the grid bounding box corresponding to each buoy.

4. The method according to claim 1, characterized in that, After the step of determining, based on the buoy movement identification result, that a buoy movement identification result corresponds to a buoy that has moved and the movement exceeds a preset movement distance threshold, and then acquiring the corresponding buoy images to form a buoy anomaly identification image set, the method further includes: If it is determined that the similarity between the buoy movement recognition results corresponding to any two buoy images in the buoy anomaly recognition image set exceeds a preset similarity threshold, then one buoy anomaly recognition image is randomly selected from the buoy anomaly recognition image set and retained to update the buoy anomaly recognition image set.

5. A device for identifying anomalies in sea surface buoys based on image recognition, characterized in that, include: The sea surface buoy image set acquisition unit is used to acquire the corresponding sea surface buoy image set in response to the sea surface buoy identification command; The set of sea surface buoy images is obtained by the image acquisition device facing the seawater inlet area with multiple sea surface buoys; The sea surface buoy identification result acquisition unit is used to acquire the sea surface buoy identification result corresponding to each sea surface buoy image in the sea surface buoy image set based on a pre-trained multi-object detection model; the acquisition of the sea surface buoy identification result corresponding to each sea surface buoy image based on the pre-trained multi-object detection model includes: determining the classification category, grid bounding box coordinates, and classification category confidence of each sea surface buoy image through the detection head in the multi-object detection model; each sea surface buoy corresponds to one image grid; The buoy distribution location area acquisition unit is used to acquire the buoy distribution location area corresponding to the buoy identification result of each buoy image; the acquisition of the buoy distribution location area corresponding to the buoy identification result of each buoy image includes: after acquiring the buoy image distribution location information of all buoys included in a buoy image, summarizing and statistically analyzing the information according to the ascending order of the unique numbers of the buoys to obtain the buoy distribution location area of ​​the buoy image; The buoy movement identification result acquisition unit is used to compare the distribution area of ​​the sea surface buoy in each sea surface buoy image with the pre-stored initial distribution area of ​​the sea surface buoy, and obtain the sea surface buoy movement identification result corresponding to each sea surface buoy image. The sea surface buoy anomaly identification unit is used to obtain the corresponding sea surface buoy images to form a sea surface buoy anomaly identification image set if it is determined based on the sea surface buoy movement identification result that the sea surface buoy movement identification result corresponds to the sea surface buoy having moved and the movement exceeding a preset movement distance threshold. An abnormal early warning information sending unit is used to generate an abnormal early warning information for sea surface buoys based on the abnormal identification image set of sea surface buoys, and send the abnormal early warning information for sea surface buoys to the user terminal; The buoy movement identification result acquisition unit is specifically used for: Obtain the coordinates of the center point of each sea surface buoy distribution area within the aforementioned sea surface buoy distribution location area; Obtain the initial coordinates of the center point of each sea surface buoy distribution area within the initial distribution location region of the sea surface buoys; The movement distance of each buoy in the distribution area is calculated by comparing the coordinates of the center point of each buoy distribution area with the initial coordinates of the center point of the corresponding buoy distribution area in the initial distribution area of ​​the buoys, thus obtaining the movement identification sub-result of each buoy in the distribution area of ​​the buoys; wherein, the movement identification sub-result of each buoy in the distribution area of ​​the buoys is the image movement distance of each buoy. Multiply the movement identification sub-result of each sea surface buoy in the distribution area of ​​the sea surface buoy by the pre-stored size ratio to obtain the actual movement identification sub-result of each sea surface buoy, and combine them to form the sea surface buoy movement identification result; The image recognition-based surface buoy anomaly detection device also includes: The actual movement identification sub-result determination unit is used to obtain the actual movement identification sub-result of each sea surface buoy based on the sea surface buoy movement identification result; The first determination unit is used to determine that if the actual movement identification result of a sea surface buoy is greater than a preset actual movement distance threshold, the movement identification result of the corresponding sea surface buoy corresponds to the sea surface buoy having moved and the movement exceeding the preset movement distance threshold, and to determine that the towing cable connecting the corresponding sea surface buoy to other sea surface buoys has broken. The second determination unit is used to determine that if the actual movement identification result of a sea surface buoy is less than or equal to a preset actual movement distance threshold, the movement identification result of the corresponding sea surface buoy corresponds to the sea surface buoy having moved and the movement not exceeding the preset movement distance threshold. The abnormal warning information sending unit is also used for: If, based on the multi-target detection model, it is determined that the sea surface buoy anomaly identification image in the sea surface buoy anomaly identification image set has a grid bounding box corresponding to a sea surface floating object classification type, and the ratio of the total area of ​​the grid bounding box of the sea surface floating object classification type to the image area of ​​the sea surface buoy anomaly identification image exceeds a preset area ratio threshold, then a sea surface floating object anomaly warning information is generated and sent to the user terminal.

6. A computer device, characterized in that, The computer device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the image recognition-based method for identifying anomalies in sea surface buoys as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which includes program instructions that, when executed by a processor, can implement the image recognition-based method for identifying anomalies in sea surface buoys as described in any one of claims 1-4.

Images