Abnormal phishing behavior alarm method and device, electronic equipment and storage medium

Abstract

The invention provides an abnormal fishing behavior warning method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining target image data, inputting the target image data into a target detection model, and obtaining a first detection result; when the first detection result represents that the abnormal fishing behavior exists in the target image data, inputting the target image data into a multi-modal model to obtain a second detection result; and when the second detection result represents that the abnormal fishing behavior exists in the target image data, giving an alarm for the abnormal fishing behavior in the target image data. Through the embodiment of the invention, the accuracy of the alarm result can be further improved by virtue of the output capability and the super generalization capability of the multi-modal model in the open domain. In addition, while a large number of invalid samples are reduced, the consumption of computing power is reduced, and the cost is saved.

Description

Technical Field

[0001] The present invention relates to the technical field of detection, and in particular to a method for warning of abnormal fishing behavior, a device for warning of abnormal fishing behavior, an electronic device and a computer-readable storage medium. Background Art

[0002] In order to detect abnormal fishing behavior, such as fishing in areas where fishing is not allowed, such training data can be collected to train a large model; however, high-quality fishing training data is limited, which often results in low accuracy and generalization performance of the target detection network. In the absence of sufficient training data, the model is prone to overfitting, resulting in poor performance in practical applications. Summary of the invention

[0003] In view of the above problems, a method for warning abnormal fishing behavior, a device for warning abnormal fishing behavior, an electronic device and a computer-readable storage medium are proposed to overcome the above problems or at least partially solve the above problems, including:

[0004] A method for warning of abnormal fishing behavior, the method comprising:

[0005] Acquire target image data, and input the target image data into a target detection model to obtain a first detection result;

[0006] When the first detection result indicates that abnormal fishing behavior exists in the target image data, inputting the target image data into a multimodal model to obtain a second detection result;

[0007] When the second detection result indicates that abnormal fishing behavior exists in the target image data, an alarm is issued for the abnormal fishing behavior in the target image data.

[0008] Optionally, acquiring target image data includes:

[0009] Acquire original input image data, and preprocess the original input image data;

[0010] Target image data is extracted from the pre-processed original input image data.

[0011] Optionally, acquiring target image data includes:

[0012] Acquire original input video data, and preprocess the original input video data;

[0013] Target image data is extracted from the pre-processed original input video data.

[0014] Optionally, the first detection result includes a first target frame of a person fishing, a second target frame of a person, a third target frame of a water area, a fourth target frame of a fishing rod, and a fifth target frame of a fish in the target image data, and target scores corresponding to the first target frame, the second target frame, the third target frame, the fourth target frame, and the fifth target frame respectively;

[0015] The method further comprises:

[0016] When the target score of the third target frame is greater than a third threshold, determining whether the first detection result meets any one of the following first conditions; when any one of the following first conditions is met, determining that the first detection result indicates that abnormal fishing behavior exists in the target image data;

[0017] The first condition is:

[0018] The target score of the second target frame is greater than a second threshold, and the target score of the fourth target frame is greater than a fourth threshold;

[0019] The target score of the second target frame is greater than the second threshold, and the target score of the fifth target frame is greater than the fifth threshold;

[0020] The target score of the first target box is greater than the first threshold.

[0021] Optionally, the method further comprises:

[0022] When the target score of the third target frame is not greater than the third threshold, the target image data is reacquired.

[0023] Optionally, the second detection result includes a sixth target frame of a person fishing, a seventh target frame of a person, an eighth target frame of a water area, a ninth target frame of a fishing rod, and a tenth target frame of a fish in the target image data, and target scores corresponding to the sixth target frame, the seventh target frame, the eighth target frame, the ninth target frame, and the tenth target frame, respectively, and image description information;

[0024] The method further comprises:

[0025] When the image description information indicates that an abnormal fishing event exists in the target image data, determining whether the second detection result meets any one of the following second conditions; when any one of the following second conditions is met, determining that the second detection result indicates that an abnormal fishing behavior exists in the target image data;

[0026] The second condition is:

[0027] The target score of the seventh target frame is greater than a seventh threshold, and the target score of the ninth target frame is greater than a ninth threshold;

[0028] The target score of the seventh target frame is greater than the seventh threshold, and the target score of the tenth target frame is greater than the tenth threshold;

[0029] The target score of the sixth target box is greater than the sixth threshold.

[0030] Optionally, the method further comprises:

[0031] When the image description information indicates that there is no abnormal fishing event in the target image data, the target image data is reacquired.

[0032] The embodiment of the present invention further provides an abnormal fishing behavior warning device, the device comprising:

[0033] A first detection module, used to obtain target image data and input the target image data into a target detection model to obtain a first detection result;

[0034] A second detection module, configured to input the target image data into a multimodal model to obtain a second detection result when the first detection result indicates that abnormal fishing behavior exists in the target image data;

[0035] An alarm module is used to issue an alarm for abnormal fishing behavior in the target image data when the second detection result indicates that abnormal fishing behavior exists in the target image data.

[0036] Optionally, the first detection module is used to obtain original input image data and preprocess the original input image data; and extract target image data from the preprocessed original input image data.

[0037] Optionally, the first detection module is used to obtain original input video data and preprocess the original input video data; and extract target image data from the preprocessed original input video data.

[0038] Optionally, the first detection result includes a first target frame of a person fishing, a second target frame of a person, a third target frame of a water area, a fourth target frame of a fishing rod, and a fifth target frame of a fish in the target image data, and target scores corresponding to the first target frame, the second target frame, the third target frame, the fourth target frame, and the fifth target frame respectively;

[0039] The first detection module is also used to determine whether the first detection result meets any one of the following first conditions when the target score of the third target box is greater than a third threshold; when any one of the following first conditions is met, determine that the first detection result represents the presence of abnormal fishing behavior in the target image data; the first condition is: the target score of the second target box is greater than the second threshold, and the target score of the fourth target box is greater than the fourth threshold; the target score of the second target box is greater than the second threshold, and the target score of the fifth target box is greater than the fifth threshold; the target score of the first target box is greater than the first threshold.

[0040] Optionally, the first detection module is further configured to reacquire the target image data when the target score of the third target frame is not greater than the third threshold.

[0041] Optionally, the second detection result includes a sixth target frame of a person fishing, a seventh target frame of a person, an eighth target frame of a water area, a ninth target frame of a fishing rod, and a tenth target frame of a fish in the target image data, and target scores corresponding to the sixth target frame, the seventh target frame, the eighth target frame, the ninth target frame, and the tenth target frame, respectively, and image description information;

[0042] The second detection module is also used to determine whether the second detection result meets any one of the following second conditions when the image description information indicates that there is an abnormal fishing event in the target image data; when any one of the following second conditions is met, it is determined that the second detection result indicates that there is abnormal fishing behavior in the target image data; the second condition is: the target score of the seventh target box is greater than the seventh threshold, and the target score of the ninth target box is greater than the ninth threshold; the target score of the seventh target box is greater than the seventh threshold, and the target score of the tenth target box is greater than the tenth threshold; the target score of the sixth target box is greater than the sixth threshold.

[0043] Optionally, the second detection module is further used to reacquire the target image data when the image description information indicates that there is no abnormal fishing event in the target image data.

[0044] An embodiment of the present invention also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the above-mentioned method for warning of abnormal fishing behavior.

[0045] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned abnormal fishing behavior alarm method is implemented.

[0046] The embodiments of the present invention have the following advantages:

[0047] In an embodiment of the present invention, target image data is obtained, and the target image data is input into a target detection model to obtain a first detection result; when the first detection result indicates that there is abnormal fishing behavior in the target image data, the target image data is input into a multimodal model to obtain a second detection result; when the second detection result indicates that there is abnormal fishing behavior in the target image data, an alarm is issued for the abnormal fishing behavior in the target image data. Through an embodiment of the present invention, target detection can be performed based on a target detection model first; when abnormal fishing behavior is detected, a secondary judgment is made on the abnormal fishing event result based on a multimodal model. With the output capability and super generalization capability of the multimodal model in the open domain, the accuracy of the alarm result is further improved. In addition, the target detection model and discrimination logic with low computational complexity can filter out most of the invalid samples that do not contain abnormal fishing events. While reducing a large number of invalid samples, the consumption of computing power is reduced, saving costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0049] Figure 1 This is a flowchart of the steps of a method for warning of abnormal fishing behavior according to an embodiment of the present invention;

[0050] Figure 2 is a flowchart of another method for warning of abnormal fishing behavior according to an embodiment of the present invention;

[0051] Figure 3 This is a flowchart of the steps of detecting abnormal fishing behavior and issuing an alarm according to an embodiment of the present invention;

[0052] Figure 4 It is a structural schematic diagram of an abnormal fishing behavior alarm device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0053] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] In order to utilize a small amount of sample data and significantly improve the accuracy of abnormal fishing target detection tasks while responding quickly, an embodiment of the present invention provides an alarm method for abnormal fishing behavior, which can first perform target detection based on a target detection model; when abnormal fishing behavior is detected, a secondary judgment is made on the abnormal fishing event results based on a multimodal model. Relying on the output capability of the multimodal model in the open domain and its super generalization ability, the accuracy of the alarm results is further improved. In addition, the target detection model and discrimination logic with low computational complexity can filter out most of the invalid samples that do not contain abnormal fishing events. While reducing a large number of invalid samples, the consumption of computing power is reduced, saving costs.

[0055] For details, please refer to Figure 1 , Figure 1 A method for warning of abnormal fishing behavior according to an embodiment of the present invention is shown.

[0056] like Figure 1 As shown, the alarm method for abnormal fishing behavior may include the following steps:

[0057] Step 101: Acquire target image data, and input the target image data into a target detection model to obtain a first detection result.

[0058] First, target image data can be obtained from a data source. The target image data can be extracted from data collected by monitoring equipment deployed around waters where fishing is not allowed. The embodiment of the present invention does not limit this. If someone is detected fishing in the target image data, it can be determined that abnormal fishing behavior exists in the target image data.

[0059] After obtaining the target image data, the alarm system can input the target image data into a pre-trained target detection model, which can be used to detect whether there are target boxes of objects related to abnormal fishing behavior in the target image data, as well as the scores of each target box.

[0060] Exemplarily, objects related to abnormal fishing behavior may be a fisherman, a fishing rod, a body of water, a fish, a person, etc., which is not limited in this embodiment of the present invention.

[0061] In practical applications, the target detection model can be trained using image data containing people fishing, or image data containing fishing rods, or image data containing water areas, or image data containing fish, or image data containing people, or image data containing a combination of the above. This enables the target detection model to have the ability to identify people fishing, fishing rods, water areas, fish, and people, and to output target frames corresponding to people fishing, fishing rods, water areas, fish, and people, as well as the scores of each target frame.

[0062] Among them, the target box can be used to locate and mark the object or area of ​​interest in the image, and the score of the target box can be used to indicate the score of the object in the area corresponding to the target box in the image as the object of interest. The higher the score, the greater the probability that the object in the area is the object of interest. The embodiment of the present invention is not limited to this.

[0063] Step 102: When the first detection result indicates that abnormal fishing behavior exists in the target image data, the target image data is input into the multimodal model to obtain a second detection result.

[0064] If the first detection result indicates that abnormal fishing behavior exists in the target image data, the alarm system may further input the target image data into the multimodal model.

[0065] In actual applications, in addition to inputting target image data into the multimodal model, question text can also be input into the multimodal model. For example, you can input "Is there any fishing behavior in this image?" into the multimodal model.

[0066] The multimodal model can detect and identify the target image data based on the input target image data and question text; specifically, the multimodal model can detect whether there are target boxes of objects related to abnormal fishing behavior in the target image data, as well as the scores of each target box, just like the target detection model.

[0067] In addition, the multimodal model can also generate a corresponding description based on the detection results and the question text; based on the description and the target box, as well as the scores of each target box, the multimodal model can output a second detection result.

[0068] Step 103: When the second detection result indicates that abnormal fishing behavior exists in the target image data, an alarm is issued for the abnormal fishing behavior in the target image data.

[0069] In actual applications, if the second detection result still indicates that there is abnormal fishing behavior in the target image data, the alarm system can determine that there is abnormal fishing behavior in the target image data; at this time, the alarm system can issue an alarm for the abnormal fishing behavior in the target image data.

[0070] Exemplarily, the alarm system may first determine the location targeted by the target image data; then, it may alert relevant personnel so that they can go to the location to correct the abnormal fishing behavior.

[0071] Of course, the alarm system can also determine the reminder devices deployed at the location targeted by the target image data, such as: image reminder devices, voice reminder devices, light reminder devices, etc.; after determining these devices, the alarm system can control these devices to remind people who engage in abnormal fishing behavior to prevent these people from continuing to engage in abnormal fishing behavior.

[0072] Exemplarily, the image reminder device may play the corresponding reminder text, the voice reminder device may play the corresponding reminder voice, and the light reminder device may flash the corresponding light information, which is not limited in this embodiment of the present invention.

[0073] In an embodiment of the present invention, target image data is obtained, and the target image data is input into a target detection model to obtain a first detection result; when the first detection result indicates that there is abnormal fishing behavior in the target image data, the target image data is input into a multimodal model to obtain a second detection result; when the second detection result indicates that there is abnormal fishing behavior in the target image data, an alarm is issued for the abnormal fishing behavior in the target image data. Through an embodiment of the present invention, target detection can be performed based on a target detection model first; when abnormal fishing behavior is detected, a secondary judgment is made on the abnormal fishing event result based on a multimodal model. With the output capability and super generalization capability of the multimodal model in the open domain, the accuracy of the alarm result is further improved. In addition, the target detection model and discrimination logic with low computational complexity can filter out most of the invalid samples that do not contain abnormal fishing events. While reducing a large number of invalid samples, the consumption of computing power is reduced, saving costs.

[0074] Reference Figure 2 , shows a flowchart of another method for warning of abnormal fishing behavior according to an embodiment of the present invention, which may include the following steps:

[0075] Step 201: Acquire target image data, and input the target image data into a target detection model to obtain a first detection result.

[0076] In practical applications, the target image data may be first obtained from a data source. After obtaining the target image data, the alarm system may input the target image data into a pre-trained target detection model, which may be used to detect whether there are target frames of objects related to abnormal fishing behavior in the target image data, as well as the scores of each target frame.

[0077] In one embodiment of the present invention, the target image data may be obtained through the following sub-steps:

[0078] Sub-step 11: Obtain original input image data and pre-process the original input image data.

[0079] In some feasible embodiments, the alarm system may be provided with an image input submodule, and the image input submodule may be responsible for receiving a static image as input; specifically, the image input submodule may first obtain static original input image data from an image acquisition device.

[0080] After obtaining the original input image data, the alarm system first pre-processes it, such as denoising and sorting.

[0081] Sub-step 12: extracting target image data from the pre-processed original input image data.

[0082] The alarm system can also be provided with a sampling strategy submodule, which can select a suitable sampling strategy according to application requirements, such as random sampling, interval sampling or key frame sampling, and extract the target image data from the preprocessed original input image data by random sampling, interval sampling or key frame sampling.

[0083] In one embodiment of the present invention, the target image data may also be acquired through the following sub-steps:

[0084] Sub-step 21: Obtain original input video data, and pre-process the original input video data.

[0085] In some feasible embodiments, the alarm system may be provided with a video input submodule, and the video input submodule may be responsible for receiving dynamic video as input; specifically, the video input submodule may first obtain dynamic original input video data from a video acquisition device.

[0086] After obtaining the original input video data, the alarm system first pre-processes it, such as denoising and sorting.

[0087] Sub-step 22: extracting target image data from the pre-processed original input video data.

[0088] The sampling strategy submodule can select an appropriate sampling strategy according to application requirements, such as random sampling, interval sampling or key frame sampling, and extract the target image data from the preprocessed original input video data by random sampling, interval sampling or key frame sampling.

[0089] Step 202, the first detection result includes a first target frame of a fishing person, a second target frame of a person, a third target frame of a water area, a fourth target frame of a fishing rod and a fifth target frame of a fish in the target image data, and target scores corresponding to the first target frame, the second target frame, the third target frame, the fourth target frame and the fifth target frame respectively; when the target score of the third target frame is greater than a third threshold, determine whether the first detection result meets any one of the following first conditions; when any one of the following first conditions is met, determine that the first detection result indicates that there is abnormal fishing behavior in the target image data.

[0090] In some feasible embodiments, the first detection result may include a first target frame of a fishing person, a second target frame of a person, a third target frame of a water area, a fourth target frame of a fishing rod, and a fifth target frame of a fish in the target image data; the first detection result may also include target scores corresponding to the first target frame, the second target frame, the third target frame, the fourth target frame, and the fifth target frame.

[0091] After obtaining the first detection result, it can be determined whether there is a water area. Specifically, the target score of the third target frame and the third threshold value can be compared.

[0092] If the target score of the third target frame is greater than the third threshold, it can be determined that there is a water target in the target image data; at this time, it can be further determined whether the first detection result meets any one of the preset first conditions.

[0093] The first condition may be:

[0094] The target score of the second target frame is greater than the second threshold, and the target score of the fourth target frame is greater than the fourth threshold;

[0095] The target score of the second target frame is greater than the second threshold, and the target score of the fifth target frame is greater than the fifth threshold;

[0096] The object score of the first object box is greater than a first threshold.

[0097] In actual applications, when the target score of the second target box is greater than the second threshold, and the target score of the fourth target box is greater than the fourth threshold; or, when the target score of the second target box is greater than the second threshold, and the target score of the fifth target box is greater than the fifth threshold; or, when the target score of the first target box is greater than the first threshold, it can be determined that the first detection result meets any one of the preset first conditions.

[0098] When it is further determined that the first detection result meets any one of the preset first conditions, it can be determined that the first detection result indicates that abnormal fishing behavior exists in the target image data.

[0099] In an embodiment of the present invention, the above method may further include the following steps:

[0100] When the target score of the third target frame is not greater than the third threshold, the target image data is reacquired.

[0101] In some feasible embodiments, if the target score of the third target frame is not greater than the third threshold, the target image data may be reacquired to continue detecting abnormal fishing behavior.

[0102] Step 203: When the first detection result indicates that abnormal fishing behavior exists in the target image data, the target image data is input into the multimodal model to obtain a second detection result.

[0103] In some feasible embodiments, when the first detection result indicates that abnormal fishing behavior exists in the target image data, the alarm system may further input the target image data into a multimodal model and obtain a second detection result output by the multimodal model.

[0104] Step 204, the second detection result includes the sixth target frame of the fisherman, the seventh target frame of the person, the eighth target frame of the water area, the ninth target frame of the fishing rod and the tenth target frame of the fish in the target image data, as well as the target scores corresponding to the sixth target frame, the seventh target frame, the eighth target frame, the ninth target frame and the tenth target frame, and the image description information; when the image description information indicates that there is an abnormal fishing event in the target image data, it is determined whether the second detection result meets any one of the following second conditions; when any one of the following second conditions is met, it is determined that the second detection result indicates that there is abnormal fishing behavior in the target image data.

[0105] Among them, the second condition is:

[0106] The target score of the seventh target frame is greater than the seventh threshold, and the target score of the ninth target frame is greater than the ninth threshold;

[0107] The target score of the seventh target frame is greater than the seventh threshold, and the target score of the tenth target frame is greater than the tenth threshold;

[0108] The target score of the sixth target box is greater than the sixth threshold.

[0109] In some feasible embodiments, the second detection result may include a sixth target frame of a person fishing, a seventh target frame of a person, an eighth target frame of water area, a ninth target frame of a fishing rod, and a tenth target frame of a fish in the target image data.

[0110] In addition, the second detection result may also include target scores corresponding to the sixth target frame, the seventh target frame, the eighth target frame, the ninth target frame, and the tenth target frame, respectively.

[0111] In addition, the second detection result may include image description information, that is, description information for the target image data, which may correspond to the input question text; for example: the question text is "Please describe the content of the target image data". When there is abnormal fishing behavior in the target image data, the image description information may be "The image shows a riverbank scene that may involve abnormal fishing. In the picture, several anglers are fishing by the river. Some of them are sitting on small chairs, holding fishing rods, while others are standing on the shore. The scene appears to be relatively quiet, the weather is gloomy, and the river is calm."

[0112] After obtaining the second detection result, it can be determined whether the image description information indicates that there is an abnormal fishing event in the target image data. For example, it can be determined whether the image description information describes the time of abnormal fishing; if so, it is determined that the image description information indicates that there is an abnormal fishing event in the target image data.

[0113] When it is determined that the image description information indicates that an abnormal fishing event exists in the target image data, it can be further determined whether the second detection result meets any one of a plurality of preset second conditions.

[0114] In actual applications, when the target score of the seventh target box is greater than the seventh threshold, and the target score of the ninth target box is greater than the ninth threshold; or when the target score of the seventh target box is greater than the seventh threshold, and the target score of the tenth target box is greater than the tenth threshold; or when the target score of the sixth target box is greater than the sixth threshold, it can be determined that the first detection result meets any one of the preset first conditions.

[0115] When it is further determined that the second detection result meets any one of the preset second conditions, it can be determined that the second detection result indicates that abnormal fishing behavior exists in the target image data.

[0116] In an embodiment of the present invention, the above method may further include the following steps:

[0117] When the image description information indicates that there is no abnormal fishing event in the target image data, the target image data is reacquired.

[0118] In some feasible embodiments, if the image description information indicates that there is no abnormal fishing event in the target image data, the target image data may be reacquired to continue detecting abnormal fishing behavior.

[0119] Step 205: When the second detection result indicates that abnormal fishing behavior exists in the target image data, an alarm is issued for the abnormal fishing behavior in the target image data.

[0120] In actual applications, if the second detection result still indicates that there is abnormal fishing behavior in the target image data, the alarm system can determine that there is abnormal fishing behavior in the target image data; at this time, the alarm system can issue an alarm for the abnormal fishing behavior in the target image data.

[0121] Exemplarily, the alarm system may first determine the location targeted by the target image data; then, it may alert relevant personnel so that they can go to the location to correct the abnormal fishing behavior.

[0122] Of course, the alarm system can also determine the reminder devices deployed at the location targeted by the target image data, such as: image reminder devices, voice reminder devices, light reminder devices, etc.; after determining these devices, the alarm system can control these devices to remind people who engage in abnormal fishing behavior to prevent these people from continuing to engage in abnormal fishing behavior.

[0123] Exemplarily, the image reminder device may play the corresponding reminder text, the voice reminder device may play the corresponding reminder voice, and the light reminder device may flash the corresponding light information, which is not limited in this embodiment of the present invention.

[0124] For example, the above alarm method can be applied to monitoring marine protected areas, where cameras and drones are deployed to monitor illegal fishing activities in real time. Through automatic detection and alarm by the system, illegal activities can be discovered and stopped in time to protect the marine ecological environment.

[0125] The above alarm method can also be applied to lake and river management, by installing monitoring equipment around lakes and rivers and combining video streams for comprehensive analysis. Multimodal large models can be used to identify illegal fishing behaviors and support the law enforcement work of local management departments.

[0126] The above alarm method can also be applied to install high-resolution cameras in aquaculture areas to monitor the impact of illegal fishing on the aquaculture environment. Real-time alarms and data analysis help managers take measures to ensure the normal operation of the aquaculture industry.

[0127] For example, Figure 3 As shown, after acquiring the target image data, it can be first input into the target detection model to obtain a first detection result; wherein the first detection result includes a first target frame of the fisherman, a second target frame of the person, a third target frame of the water area, a fourth target frame of the fishing rod and a fifth target frame of the fish, as well as target scores corresponding to the first target frame, the second target frame, the third target frame, the fourth target frame and the fifth target frame.

[0128] Based on the first detection result, it is possible to first determine whether there is water area in the target image data; if there is no water area, no alarm is given; if there is water area, it is possible to further determine whether there is a person and a fishing rod, or a person and a fish, or a person fishing.

[0129] If there are no person, fishing rod, fish and fisherman, no alarm is given. If there are a person and a fishing rod, or a person and a fish, or a fisherman, the target image data can be further input into the multimodal model to obtain a second detection result.

[0130] Among them, the second detection result includes the sixth target frame of the fisherman, the seventh target frame of the person, the eighth target frame of the water area, the ninth target frame of the fishing rod and the tenth target frame of the fish, as well as the target scores corresponding to the sixth target frame, the seventh target frame, the eighth target frame, the ninth target frame and the tenth target frame, and image description information.

[0131] Based on the image description information of the second detection result, it can be determined whether there is an abnormal fishing event in the target image data; if not, no alarm is given. If so, it can be further determined whether there is a person and a fish, or a person and a fishing rod, or a person fishing.

[0132] If there are no people, fish, fishing rods and fishermen, no alarm will be triggered; if there are people and fish, or people and fishing rods, or fishermen, an alarm may be triggered.

[0133] In an embodiment of the present invention, target image data is acquired and input into a target detection model to obtain a first detection result; the first detection result includes a first target frame of a fishing person, a second target frame of a person, a third target frame of a body of water, a fourth target frame of a fishing rod, and a fifth target frame of a fish in the target image data, as well as target scores corresponding to the first target frame, the second target frame, the third target frame, the fourth target frame, and the fifth target frame; when the target score of the third target frame is greater than a third threshold, it is determined whether the first detection result meets any one of the following first conditions; when any one of the following first conditions is met, it is determined that the first detection result indicates that there is abnormal fishing behavior in the target image data; when the first detection result indicates that there is abnormal fishing behavior in the target image data, the target image data is input into a target detection model. A multimodal model is used to obtain a second detection result; the second detection result includes a sixth target frame of a fishing person, a seventh target frame of a person, an eighth target frame of a water area, a ninth target frame of a fishing rod and a tenth target frame of a fish in the target image data, as well as target scores corresponding to the sixth target frame, the seventh target frame, the eighth target frame, the ninth target frame and the tenth target frame, and image description information; when the image description information indicates that there is an abnormal fishing event in the target image data, it is determined whether the second detection result meets any one of the following second conditions; when any one of the following second conditions is met, it is determined that the second detection result indicates that there is an abnormal fishing behavior in the target image data; when the second detection result indicates that there is an abnormal fishing behavior in the target image data, an alarm is issued for the abnormal fishing behavior in the target image data.

[0134] Through the embodiments of the present invention, target detection can be performed based on the target detection model first; when abnormal fishing behavior is detected, the abnormal fishing event result can be secondarily judged based on the analysis of the multimodal model. With the output capability of the multimodal model in the open domain and the super generalization capability, the accuracy of the alarm result is further improved. In addition, the target detection model and discrimination logic with low computational complexity can filter out most of the invalid samples that do not contain abnormal fishing events. While reducing a large number of invalid samples, the consumption of computing power is reduced, saving costs.

[0135] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0136] Reference Figure 4 , shows a schematic diagram of the structure of an abnormal fishing behavior alarm device according to an embodiment of the present invention, which may include the following modules:

[0137] The first detection module 401 is used to obtain target image data and input the target image data into a target detection model to obtain a first detection result;

[0138] A second detection module 402 is configured to input the target image data into the multimodal model to obtain a second detection result when the first detection result indicates that abnormal fishing behavior exists in the target image data;

[0139] The alarm module 403 is configured to generate an alarm for the abnormal fishing behavior in the target image data when the second detection result indicates that the target image data contains abnormal fishing behavior.

[0140] In an optional embodiment of the present invention, the first detection module 401 is used to obtain original input image data and preprocess the original input image data; and extract target image data from the preprocessed original input image data.

[0141] In an optional embodiment of the present invention, the first detection module 401 is used to obtain original input video data and preprocess the original input video data; and extract target image data from the preprocessed original input video data.

[0142] In an optional embodiment of the present invention, the first detection result includes a first target frame of a person fishing, a second target frame of a person, a third target frame of a water area, a fourth target frame of a fishing rod, and a fifth target frame of a fish in the target image data, and target scores corresponding to the first target frame, the second target frame, the third target frame, the fourth target frame, and the fifth target frame respectively;

[0143] The first detection module 401 is also used to determine whether the first detection result meets any one of the following first conditions when the target score of the third target box is greater than the third threshold; when any one of the following first conditions is met, it is determined that the first detection result represents the presence of abnormal fishing behavior in the target image data; the first condition is: the target score of the second target box is greater than the second threshold, and the target score of the fourth target box is greater than the fourth threshold; the target score of the second target box is greater than the second threshold, and the target score of the fifth target box is greater than the fifth threshold; the target score of the first target box is greater than the first threshold.

[0144] In an optional embodiment of the present invention, the first detection module 402 is further configured to reacquire the target image data when the target score of the third target frame is not greater than a third threshold.

[0145] In an optional embodiment of the present invention, the second detection result includes a sixth target frame of a person fishing, a seventh target frame of a person, an eighth target frame of a water area, a ninth target frame of a fishing rod, and a tenth target frame of a fish in the target image data, and target scores corresponding to the sixth target frame, the seventh target frame, the eighth target frame, the ninth target frame, and the tenth target frame, respectively, and image description information;

[0146] The second detection module 402 is also used to determine whether the second detection result meets any one of the following second conditions when the image description information indicates that there is an abnormal fishing event in the target image data; when any one of the following second conditions is met, it is determined that the second detection result indicates that there is an abnormal fishing behavior in the target image data; the second condition is: the target score of the seventh target box is greater than the seventh threshold, and the target score of the ninth target box is greater than the ninth threshold; the target score of the seventh target box is greater than the seventh threshold, and the target score of the tenth target box is greater than the tenth threshold; the target score of the sixth target box is greater than the sixth threshold.

[0147] In an optional embodiment of the present invention, the second detection module 402 is further configured to reacquire the target image data when the image description information indicates that there is no abnormal fishing event in the target image data.

[0148] In an embodiment of the present invention, target image data is obtained, and the target image data is input into a target detection model to obtain a first detection result; when the first detection result indicates that there is abnormal fishing behavior in the target image data, the target image data is input into a multimodal model to obtain a second detection result; when the second detection result indicates that there is abnormal fishing behavior in the target image data, an alarm is issued for the abnormal fishing behavior in the target image data. Through an embodiment of the present invention, target detection can be performed based on a target detection model first; when abnormal fishing behavior is detected, a secondary judgment is made on the abnormal fishing event result based on a multimodal model. With the output capability and super generalization capability of the multimodal model in the open domain, the accuracy of the alarm result is further improved. In addition, the target detection model and discrimination logic with low computational complexity can filter out most of the invalid samples that do not contain abnormal fishing events. While reducing a large number of invalid samples, the consumption of computing power is reduced, saving costs.

[0149] An embodiment of the present invention also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and capable of running on the processor, and when the computer program is executed by the processor, the above-mentioned abnormal fishing behavior alarm method is implemented.

[0150] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned abnormal fishing behavior alarm method is implemented.

[0151] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0152] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0153] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0154] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0155] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0156] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0157] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0158] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0159] The above provides a detailed introduction to an abnormal fishing behavior alarm method, an abnormal fishing behavior alarm device, an electronic device and a computer-readable storage medium. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for warning of abnormal fishing behavior, characterized in that: The method comprises: Acquire target image data, and input the target image data into a target detection model to obtain a first detection result; When the first detection result indicates that abnormal fishing behavior exists in the target image data, inputting the target image data into a multimodal model to obtain a second detection result; When the second detection result indicates that abnormal fishing behavior exists in the target image data, an alarm is issued for the abnormal fishing behavior in the target image data.

2. The method according to claim 1, characterized in that The acquiring target image data comprises: Acquire original input image data, and preprocess the original input image data; Target image data is extracted from the pre-processed original input image data.

3. The method according to claim 1, characterized in that The acquiring target image data comprises: Acquire original input video data, and preprocess the original input video data; Target image data is extracted from the pre-processed original input video data.

4. The method according to claim 1, characterized in that The first detection result includes a first target frame of a person fishing, a second target frame of a person, a third target frame of a water area, a fourth target frame of a fishing rod, and a fifth target frame of a fish in the target image data, and target scores corresponding to the first target frame, the second target frame, the third target frame, the fourth target frame, and the fifth target frame respectively; The method further comprises: When the target score of the third target frame is greater than a third threshold, determining whether the first detection result meets any one of the following first conditions; when any one of the following first conditions is met, determining that the first detection result indicates that abnormal fishing behavior exists in the target image data; The first condition is: The target score of the second target frame is greater than a second threshold, and the target score of the fourth target frame is greater than a fourth threshold; The target score of the second target frame is greater than the second threshold, and the target score of the fifth target frame is greater than the fifth threshold; The target score of the first target box is greater than the first threshold.

5. The method according to claim 4, characterized in that The method further comprises: When the target score of the third target frame is not greater than the third threshold, the target image data is reacquired.

6. The method according to claim 1, characterized in that The second detection result includes a sixth target frame of a person fishing, a seventh target frame of a person, an eighth target frame of a water area, a ninth target frame of a fishing rod, and a tenth target frame of a fish in the target image data, and target scores corresponding to the sixth target frame, the seventh target frame, the eighth target frame, the ninth target frame, and the tenth target frame, respectively, and image description information; The method further comprises: When the image description information indicates that an abnormal fishing event exists in the target image data, determining whether the second detection result meets any one of the following second conditions; when any one of the following second conditions is met, determining that the second detection result indicates that an abnormal fishing behavior exists in the target image data; The second condition is: The target score of the seventh target frame is greater than a seventh threshold, and the target score of the ninth target frame is greater than a ninth threshold; The target score of the seventh target frame is greater than the seventh threshold, and the target score of the tenth target frame is greater than the tenth threshold; The target score of the sixth target box is greater than the sixth threshold.

7. The method according to claim 6, characterized in that The method further comprises: When the image description information indicates that there is no abnormal fishing event in the target image data, the target image data is reacquired.

8. An alarm device for abnormal fishing behavior, characterized in that: The device comprises: A first detection module, used to obtain target image data and input the target image data into a target detection model to obtain a first detection result; A second detection module, configured to input the target image data into a multimodal model to obtain a second detection result when the first detection result indicates that abnormal fishing behavior exists in the target image data; An alarm module is used to issue an alarm for abnormal fishing behavior in the target image data when the second detection result indicates that abnormal fishing behavior exists in the target image data.

9. An electronic device, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the method for warning of abnormal fishing behavior as claimed in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for warning of abnormal fishing behavior as described in any one of claims 1 to 7 is implemented.

Images