Abnormal behavior detection method and device for vehicles, and camera device

Abstract

The present application relates to a method and device for detecting abnormal behavior of a vehicle, and a camera device, wherein the method comprises: acquiring an image sequence of an area to be detected, wherein the image sequence comprises a plurality of continuous image frames; tracking a vehicle in the image sequence, and determining position information of the vehicle in the image sequence in the plurality of continuous image frames; determining motion state information of the vehicle based on the position information of the vehicle in the plurality of continuous image frames; and determining whether the vehicle has abnormal behavior associated with the area to be detected based on the motion state information.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method and device for detecting abnormal behavior of vehicles, and a camera device. Background Art

[0002] Vehicles may exhibit some abnormal behaviors during driving. Abnormal behaviors of vehicles may lead to traffic accidents, or even be caused by vehicles engaging in illegal activities. Therefore, rapid and accurate detection of abnormal behaviors of vehicles is crucial to maintaining traffic order, public safety and public interests.

[0003] The related art has not yet proposed a detection method for abnormal behavior of vehicles. Summary of the invention

[0004] The embodiments of the present application provide a method and device for detecting abnormal behavior of a vehicle, and a camera device, so as to at least solve the problem that the abnormal behavior of a vehicle cannot be detected in the related art.

[0005] In a first aspect, an embodiment of the present application provides a method for detecting abnormal behavior of a vehicle, comprising:

[0006] Acquire an image sequence of the area to be detected, wherein the image sequence includes a plurality of continuous image frames;

[0007] Tracking the vehicle in the image sequence to determine the position information of the vehicle in the image sequence in the plurality of consecutive image frames;

[0008] Determining motion state information of the vehicle according to position information of the vehicle in the plurality of consecutive image frames;

[0009] It is determined whether the vehicle has abnormal behavior associated with the area to be detected according to the motion state information.

[0010] Optionally, in one embodiment of the present application, the motion state information includes a stationary state;

[0011] The determining, based on the motion state information, whether the vehicle has abnormal behavior associated with the area to be detected includes:

[0012] Determining a stationary time duration of the vehicle in the stationary state in the image sequence;

[0013] Based on the stationary time duration and the stationary time duration threshold, it is determined whether the vehicle has abnormal behavior associated with the area to be detected.

[0014] Optionally, in one embodiment of the present application, determining whether the vehicle has abnormal behavior associated with the area to be detected based on the stationary time and the stationary time threshold includes:

[0015] In response to the stationary time length being greater than or equal to a stationary time length threshold, it is determined that the vehicle has abnormal behavior associated with the area to be detected.

[0016] Optionally, in an embodiment of the present application, determining whether the vehicle has abnormal behavior associated with the area to be detected according to the motion state information includes:

[0017] In response to the motion state information including at least two sub-motion states, determining a state transition trend of the at least two sub-motion states;

[0018] Based on the state transition trend, it is determined whether the vehicle has abnormal behavior associated with the area to be detected.

[0019] Optionally, in one embodiment of the present application, the at least two sub-motion states include a stationary state and a moving state;

[0020] The determining, based on the state transition trend, whether the vehicle has abnormal behavior associated with the area to be detected includes:

[0021] In response to the state transition trend being consistent with a preset state transition trend, it is determined that the vehicle has abnormal behavior associated with the area to be detected; wherein the preset state transition trend includes transitioning from the moving state to the stationary state.

[0022] Optionally, in one embodiment of the present application, the transportation vehicle includes a ship, and the abnormal behavior includes sand mining behavior.

[0023] Optionally, in one embodiment of the present application, the transportation vehicle includes a ship, and the abnormal behavior includes sand mining behavior; in response to the state transition trend being consistent with a preset state transition trend, determining that the transportation vehicle has an abnormal behavior associated with the area to be detected includes:

[0024] If it is detected that the ship in the image sequence switches from a moving state to a stationary state, then in response to a time length for which the ship is stationary being greater than a stationary time length threshold, it is determined that the ship has abnormal behavior associated with the area to be detected.

[0025] Optionally, in one embodiment of the present application, the transportation vehicle includes a ship, and the abnormal behavior includes sand mining behavior; after determining whether the transportation vehicle has abnormal behavior associated with the area to be detected according to the motion state information, the method further includes:

[0026] In response to determining that the vessel has engaged in the sand mining behavior, a target thermal imaging image is determined based on the thermal imaging image to be detected collected for the vessel; the area proportion of the vessel in the target thermal imaging is greater than an area threshold;

[0027] Detecting whether there is a heat source of a sand mining unit in the target thermal imaging image;

[0028] In response to detecting the heat source of the sand mining unit in the target thermal imaging image, it is determined that the vessel has engaged in illegal sand mining.

[0029] Optionally, in one embodiment of the present application, after detecting whether there is a heat source of a sand mining unit in the target thermal imaging image, the method further includes:

[0030] In response to the fact that no heat source of the sand mining unit is detected in the target thermal imaging image, it is determined that the vessel has not engaged in any illegal sand mining activities.

[0031] Optionally, in one embodiment of the present application, before determining the target thermal imaging image based on the to-be-detected thermal imaging image collected for the vessel, the method further includes:

[0032] The image sequence includes a plurality of continuous visible light images; a visible light image for determining the sand mining behavior is determined from the plurality of visible light images, and a thermal imaging image collected for the ship at the time of collecting the determined visible light image is obtained as the thermal imaging image to be detected; or

[0033] The image sequence includes a plurality of continuous thermal imaging images; the thermal imaging image used to determine the sand mining behavior among the plurality of thermal imaging images is used as the thermal imaging image to be detected.

[0034] Optionally, in one embodiment of the present application, determining the target thermal imaging image based on the thermal imaging image to be detected collected from the ship includes:

[0035] An image region including the vessel is acquired from the thermal imaging image to be detected and determined as the target thermal imaging image; the target thermal imaging image is a sub-image of the thermal imaging image to be detected.

[0036] Optionally, in one embodiment of the present application, the detecting whether there is a heat source of a sand mining unit in the target thermal imaging image includes:

[0037] The target thermal imaging image is detected using the trained sand mining unit detection network, and whether there is a sand mining unit heat source in the target thermal imaging image is determined based on the detection result; wherein:

[0038] The sand mining machine group detection network is obtained by training based on the thermal imaging image samples that are labeled for the sand mining machine group.

[0039] Optionally, in one embodiment of the present application, the detecting whether there is a heat source of a sand mining unit in the target thermal imaging image includes:

[0040] Binarizing the target thermal imaging image to obtain a temperature binary image;

[0041] If it is determined that a connected domain that meets preset conditions exists at a preset position in the temperature binary image, it is determined that the heat source of the sand mining unit is detected in the target thermal imaging image.

[0042] Optionally, in one embodiment of the present application, the target tracking of the vehicle in the image sequence and determining the position information of the vehicle in the image sequence in the plurality of consecutive image frames includes:

[0043] Inputting the plurality of continuous image frames into a vehicle detection model respectively, and outputting the position information and the unique identification of the vehicle in the image frames through the vehicle detection model;

[0044] According to the unique identification of the vehicle, position information of the vehicle in the multiple consecutive image frames is determined.

[0045] In a second aspect, the embodiment of the present application further provides an abnormal behavior device for a vehicle, comprising:

[0046] An image acquisition module, used to acquire an image sequence of the area to be detected, wherein the image sequence includes a plurality of continuous image frames;

[0047] A target tracking module, used to track the vehicle in the image sequence and determine the position information of the vehicle in the image sequence in the plurality of consecutive image frames;

[0048] a motion state determination module, configured to determine the motion state information of the vehicle according to the position information of the vehicle in the plurality of consecutive image frames;

[0049] The abnormal behavior determination module is used to determine whether the vehicle has abnormal behavior associated with the area to be detected based on the motion state information.

[0050] In a third aspect, a camera device includes a lens group, an image sensor, a memory and a processor, characterized in that a computer program is stored in the memory, and the processor is configured to run the computer program to execute the abnormal behavior detection method.

[0051] In a fourth aspect, a non-volatile computer-readable storage medium stores computer program instructions, which implement the abnormal behavior detection method when executed by a processor.

[0052] In a fifth aspect, a computer program product includes a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the abnormal behavior detection method.

[0053] In a sixth aspect, a chip comprises at least one processor, wherein the processor is used to run a computer program or computer instructions stored in a memory to execute the abnormal behavior detection method.

[0054] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent.

[0055] The abnormal behavior detection method for vehicles provided in various embodiments of the present application can track the vehicles in the area to be detected based on the image sequence of the area to be detected, determine the position information of the vehicles in multiple consecutive image frames in the image sequence, and determine the motion state information of the vehicles based on the position information. According to the motion state information, it can be determined whether the vehicle has abnormal behavior associated with the area to be detected. The motion state information belongs to the attributes of the vehicle during the movement process. Therefore, determining whether the vehicle has abnormal behavior based on the motion state information not only has high accuracy and detection efficiency, but also has the advantages of low cost and 24-hour all-weather detection, which meets the needs of abnormal behavior detection of vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0057] Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0058] Figure 2 is a method flow chart of a method for detecting abnormal behavior of a vehicle provided in an embodiment of the present application;

[0059] Figure 3 This is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0060] Figure 4This is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0061] Figure 5 This is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0062] Figure 6 This is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0063] Figure 7 It is a schematic diagram of the module structure of an abnormal behavior detection device for a vehicle provided in an embodiment of the present application;

[0064] Figure 8 It is a schematic diagram of the module structure of the processing device provided in the embodiment of the present application;

[0065] Fig. 9 It is a schematic diagram of the module structure of the computer program product provided in the embodiment of the present application. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application. In addition, it can also be understood that although the efforts made in this development process may be complex and lengthy, for ordinary technicians in the field related to the contents disclosed in the present application, some changes such as design, manufacturing or production based on the technical contents disclosed in the present application are only conventional technical means, and should not be understood as insufficient contents disclosed in the present application.

[0067] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0068] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to greater than or equal to two. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The terms "first", "second", "third" and the like involved in the present application are merely used to distinguish similar objects and do not represent a specific ordering of the objects.

[0069] In addition, in order to better illustrate the present application, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present application can also be implemented without certain specific details. In some examples, devices, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present application.

[0070] Based on technical requirements similar to those mentioned above, an embodiment of the present application proposes a method for detecting abnormal behavior of a vehicle, which can track the vehicle through an image sequence and determine the position information of the vehicle in each image frame. According to the position information of the vehicle in the image frame, the motion state information of the vehicle is determined. Finally, it is determined whether the vehicle has abnormal behavior based on the motion state information. Among them, the vehicle is not limited to the above-mentioned ships, but can also include cars, bicycles, automobiles, motorcycles, trains, aircraft, etc. The present application does not limit the type of vehicle.

[0071] In order to facilitate understanding of the embodiments of the present application, the structure of one of the abnormal behavior detection systems for vehicles based on the embodiments of the present application is described below. Figure 1 , Figure 1It is a structural diagram of an abnormal behavior detection system provided in an embodiment of the present application, wherein the system includes a collection device 101 and an abnormal behavior detection device 103, wherein the collection device 101 and the abnormal behavior detection device 103 can communicate via a network to send the collected images for abnormal behavior detection to the abnormal behavior detection device 103, and the abnormal behavior detection device 103 completes the detection of abnormal behavior of a vehicle.

[0072] The acquisition device 101 may be an electronic device with image acquisition and data transmission capabilities. For example, the acquisition device 101 may include a camera, a laser radar, etc., wherein the camera may include a gun-type camera, a pan-tilt camera, a thermal imaging camera, a binocular camera, a depth camera, etc., and the laser radar may include a single-line radar, a multi-line radar, etc., and the present application does not limit the acquisition device 101. The acquisition device 101 may also include a roadside unit, which may be installed on both sides of the road or on the shore or in the waters of the waters with monitoring requirements. It should be noted that the image of the area to be detected can be collected by one roadside unit, or multiple roadside units may cooperate to collect the image of the area to be detected. Among them, the data transmission function of the acquisition device 101 can be realized by a high-gain directional beam-controlled read-write antenna and a radio frequency controller. The high-gain directional beam-controlled read-write antenna is a microwave transceiver module, which is responsible for sending / receiving, modulating / demodulating, encoding / decoding, encrypting / decrypting signals and data; the radio frequency controller is a module that controls the transmission and reception of data and processes the information sent and received to the upper computer.

[0073] The abnormal behavior detection device 103 can be an electronic device with data processing capability and data receiving and sending capability, and can be a physical device such as a host, a rack server, a blade server, etc., or a virtual device such as a virtual machine, a container, etc. After acquiring the image sequence of the area to be detected, the abnormal behavior detection device 103 can track the ship in the image sequence and determine the position information of the ship in the image sequence in the multiple continuous image frames. Then, according to the position information of the ship in the multiple continuous image frames, the motion feature information of the ship is determined. Finally, it can be determined whether the ship has sand mining behavior according to the motion feature information. In addition to electronic devices, the abnormal behavior detection device 103 can also include various forms such as non-volatile computer-readable storage media, computer program products, chips, etc. As a non-volatile computer-readable storage medium, a computer program product, and a chip, the abnormal behavior detection device 103 can be coupled to the inside of the acquisition device 101, so that the acquisition device 101 has the function of abnormal behavior detection.

[0074] The abnormal behavior detection method for vehicles described in this application is described in detail below with reference to the accompanying drawings. Figure 2It is a flow chart of an embodiment of the abnormal behavior detection method for vehicles provided by the present application. Although the present application provides the method operation steps as shown in the following embodiments or drawings, more or fewer operation steps may be included in the method based on routine or no creative labor. In the steps where there is no necessary causal relationship logically, the execution order of these steps is not limited to the execution order provided in the embodiments of the present application. The method can be executed in the method sequence shown in the embodiments or drawings or in parallel (for example, in a parallel processor or multi-threaded processing environment) during the actual abnormal behavior detection process for vehicles or when the method is executed.

[0075] Specifically, an implementation example of the abnormal behavior detection method for a vehicle provided by the present application is as follows: Figure 2 As shown, the method may include:

[0076] S201: Acquire an image sequence of a region to be detected, wherein the image sequence includes a plurality of continuous image frames.

[0077] In an embodiment of the present application, the area to be detected may include areas where vehicles can travel, such as roads and waters with detection requirements, such as elevated roads, no parking areas, and riverbanks. The acquisition device 101 can be used to collect images or videos of the area to be detected. Depending on the different acquisition principles of the acquisition device 101, the types of images collected are also different. Optionally, in an embodiment of the present application, the acquisition device 101 may include a thermal imaging camera, and the type of image collected is an infrared thermal imaging image. The thermal imaging camera, for example, includes a thermal imaging monocular gimbal, a thermal imaging binocular gimbal, a thermal imaging monocular gun camera, a thermal imaging binocular gun camera, etc., which are not limited in this application. In the case where the area to be detected is a water environment, fog, water vapor, etc. are easily generated, and a thermal imaging camera can be used to capture clear images even in foggy weather. Of course, in other embodiments, the image type may also include visible light images, three-dimensional point cloud images, depth images, etc., which are not limited in this application. The image sequence may include multiple continuous image frames, and the multiple continuous image frames may include multiple continuous frames of images acquired according to a preset frame rate. The preset frame rate may be set to 10fps, 8fps, 1fps, etc., for example, and is not limited here.

[0078] It should be noted that the process of acquiring the image sequence in the embodiment of the present application is a continuous process, and the image sequence can be acquired once every preset time interval. For example, the acquisition device 101 can be set to upload the image or video acquired in the previous minute to the abnormal behavior detection device 103 every minute, and the abnormal behavior detection device 103 processes the uploaded image or video. Of course, the preset time interval can be set to 20s, 40s, 2min, etc., and there is no limitation here.

[0079] S203: Tracking the vehicle in the image sequence to determine position information of the vehicle in the image sequence in the plurality of consecutive image frames.

[0080] In an embodiment of the present application, after acquiring the image sequence of the area to be detected, the vehicles in the image sequence may be tracked to determine the position information of the vehicles in the image sequence in the multiple consecutive image frames. Here, the target tracking may include tracking the same vehicle in different image frames to determine the position of the vehicle in each image frame, thereby determining the travel path of the vehicle over time. In one embodiment of the present application, a machine learning model may be used to perform target detection on the multiple consecutive image frames respectively. Specifically, the target tracking of the vehicles in the image sequence to determine the position information of the vehicles in the image sequence in the multiple consecutive image frames may include:

[0081] S301: inputting the plurality of continuous image frames into a vehicle detection model respectively, and outputting the position information and the unique identification of the vehicle in the image frames through the vehicle detection model;

[0082] S303: Determine the position information of the vehicle in the plurality of consecutive image frames according to the unique identifier of the vehicle.

[0083] In the embodiment of the present application, the vehicle detection model may include a model trained by deep learning, and is used to detect the regional image of the vehicle in the image frame. Figure 3 As shown, in the case where the vehicle is a ship, Figure 3The image frame shown is input into a ship detection model, and the ship detection model can identify the regional image of the ship in the image frame, and can also use a bounding box to frame the regional image of the ship. The location information of the vehicle can include the location of the regional image of the vehicle in the image frame. In addition, the vehicle detection model can also distinguish different vehicles and set a unique vehicle identifier for the detected vehicle, and the unique vehicle identifier is used to characterize the uniqueness of the vehicle. Figure 3 As shown, the location information and unique identification of the ship can be marked next to the bounding box. Among them, "ship 1", "ship 2", and "ship 3" can be used as unique identifications of ships. Then, based on the unique identification of the vehicle, the location information of the vehicle in the multiple consecutive image frames can be determined. In one example, for example, the location information of the ship uniquely identified as "ship 1" in k consecutive image frames is determined as: (x1, y1) → (x2, y2) → (x3, y3) → ... → (x k ,y k ), the position information of other ships can also be obtained. By tracking each vehicle, the position information of each vehicle in different image frames can be determined. Figure 4 As shown, the position information of "Ship 1", "Ship 2" and "Ship 3" after k frames.

[0084] It should be noted that the vehicle detection model can be trained by a variety of target detection network frameworks. Typically, the target detection network frameworks include RCNN series (including Faster R-CNN, Mask R-CNN), SSD, YOLO series, etc. This application does not limit this.

[0085] S205: Determine the motion state information of the vehicle according to the position information of the vehicle in the multiple consecutive image frames.

[0086] S207: Determine, based on the motion state information, whether the vehicle has abnormal behavior associated with the area to be detected.

[0087] In an embodiment of the present application, the motion state information of the vehicle can be determined based on the position information of the vehicle in the multiple consecutive image frames. The motion state information may include a stationary state, a moving state, and of course may also include a state transition between a stationary state and a moving state, etc., which is not limited in the present application. Among them, the stationary state or the moving state of the vehicle can be determined based on the position difference of the vehicle in a preset number of image frames. In one example, for k image frames, k is greater than or equal to 2, then the position difference of the vehicle in these k image frames can be expressed as:

[0088]

[0089] Among them, (x k ,y k ) is the position of the vehicle in the kth image frame, and (x1, y1) is the position of the vehicle in the 1st image frame.

[0090] After determining the position difference of the vehicle within the k image frames, it is possible to determine whether the vehicle is stationary or in motion based on a comparison of the position difference with a preset displacement threshold. For example, if it is determined that the position difference is less than the preset displacement threshold, the vehicle is determined to be stationary; otherwise, the vehicle is in motion. Of course, the movement rate of the vehicle can also be determined based on the position difference and the frame rate of the image frame, and by comparing the movement rate with a preset rate, it is determined whether the vehicle is stationary or in motion. The present application does not limit the method for determining whether the vehicle is stationary or in motion.

[0091] In one embodiment of the present application, the motion state information includes a stationary state;

[0092] The determining, based on the motion state information, whether the vehicle has abnormal behavior associated with the area to be detected includes:

[0093] S401: Determine the static time duration of the vehicle in the static state in the image sequence;

[0094] S403: Based on the stationary time and the stationary time threshold, determine whether the vehicle has abnormal behavior associated with the area to be detected.

[0095] In actual scenarios, if a vehicle is stationary for a long time in the area to be detected, it can often be determined that the vehicle has abnormal behavior. For example, for a no-parking area, if a vehicle stops for a long time, it can be determined that the vehicle has abnormal behavior. For a highway area, if a vehicle is stationary for a long time, it can be determined that the vehicle has abnormal behavior on the highway. For another example, for a sand mining ship, if the ship is stationary on the water for a long time, it can be determined that the ship has the possibility of illegal sand mining. Based on this, it can be determined whether the vehicle has abnormal behavior based on the stationary duration and the stationary duration threshold when the vehicle is in a stationary state. In an embodiment of the present application, in the process of determining the stationary duration of the vehicle, the starting time when the vehicle starts to be stationary is first determined, and then the stationary duration can be accumulated from the starting time. Specifically, it can be determined once every several image frames whether the vehicle is still in a stationary state. If it is still in a stationary state, the stationary duration is continuously accumulated based on the acquisition frame rate of the image frame. In one embodiment of the present application, in response to the stationary duration being greater than or equal to the stationary duration threshold, it is determined that the vehicle has abnormal behavior associated with the area to be detected. In one example, for example, the starting time of the ship being stationary on the water surface is determined to be 10:30 am, and after a period of accumulation, the stationary time of the ship is obtained to be 50 minutes. According to the preset stationary time threshold of 40 minutes, it can be determined that the ship has engaged in sand mining.

[0096] In the embodiment of the present application, the vehicle may be allowed to move briefly during the stationary process. The brief movement of the vehicle does not affect the accumulation of the stationary time, but it is necessary to meet the condition that the time the vehicle is in a moving state is less than the time it is in a stationary state, and the determined stationary time may include the accumulated time the vehicle is in a stationary state minus the accumulated time it is in a moving state. In one example, for example, it is determined that the starting time of the ship's stationary state is 10:30 am, and after a period of accumulation, the stationary time of the ship is 20 minutes. Thereafter, it is detected that the ship starts to move, and according to the preset timing rules, the time the ship moves can be deducted from the above-mentioned stationary time. For example, the ship moves for 5 minutes and then stops again, and the deducted stationary time is 15 minutes. The stationary time of the ship continues to be accumulated until the stationary time is greater than or equal to the stationary time threshold.

[0097] In actual situations, when a vehicle has abnormal behavior, its actual motion state changes. For example, for a sand mining vessel, it may move briefly during a long period of stationary state, for example, adjusting from one position to the next position to continue sand mining. Based on this, in one embodiment of the present application, determining whether the vehicle has abnormal behavior associated with the area to be detected based on the motion state information includes:

[0098] S501: In response to the motion state information including at least two sub-motion states, determining a state transition trend of the at least two sub-motion states;

[0099] S503: Based on the state transition trend, determine whether the vehicle has abnormal behavior associated with the area to be detected.

[0100] In an embodiment of the present application, it can be determined whether the vehicle has abnormal behavior associated with the area to be detected based on the state transition trend of the vehicle. The state transition trend may include the switching of the vehicle between at least two sub-motion states. The sub-motion state may include a stationary state and a moving state. Furthermore, the moving state may also include a fast moving state, a slow moving state, etc., which is not limited in the present application. Of course, the state transition trend may include multiple transitions between the at least two sub-motion states, such as transitioning from a stationary state to a moving state, and then from a moving state to a stationary state, etc. The present application does not limit the number of the sub-motion states and the number of transitions.

[0101] In one embodiment of the present application, the at least two sub-motion states include a stationary state and a moving state;

[0102] The determining, based on the state transition trend, whether the vehicle has abnormal behavior associated with the area to be detected includes:

[0103] In response to the state transition trend being consistent with a preset state transition trend, it is determined that the vehicle has abnormal behavior associated with the area to be detected; wherein the preset state transition trend includes transitioning from the moving state to the stationary state.

[0104] In the embodiment of the present application, by setting a preset state transition trend, and according to the state transition trend, it is determined whether the vehicle has abnormal behavior associated with the area to be detected. In one embodiment, the preset state transition trend may include transitioning from the moving state to the stationary state. In other embodiments, the preset state transition trend may also include transitioning from the stationary state to the moving state, and then from the moving state to the stationary state. The present application does not limit the preset state transition trend.

[0105] Illegal sand mining has seriously threatened the ecological environment of rivers, lakes and other water bodies, the navigation safety of ships, and the flood control function of embankments and riverbanks. Therefore, the detection of sand mining has become an indispensable measure to resist. In the relevant technology, the detection of sand mining is mainly carried out by comparing the hull shape characteristics of the ship with the shape characteristics of the standard sand mining ship. After comparison, if the shape characteristics of the ship match those of the standard sand mining ship, the ship is determined to be a sand mining ship. However, according to relevant public information, many illegal sand mining ships will disguise themselves as cargo ships and passenger ships to carry out illegal sand mining. It can be seen that the detection method of sand mining in the relevant technology is likely to be missed.

[0106] In the embodiment of the present application, the transportation tool may include a ship, and the abnormal behavior may include sand mining behavior. In response to the state transition trend being consistent with a preset state transition trend, determining that the transportation tool has an abnormal behavior associated with the area to be detected may include:

[0107] If it is detected that the ship in the image sequence switches from a moving state to a stationary state, then in response to a time length for which the ship is stationary being greater than a stationary time length threshold, it is determined that the ship has abnormal behavior associated with the area to be detected.

[0108] In actual application scenarios, ships may also remain still on the water for a long time when they are anchored or stranded. Further, the applicant found in the process of studying the characteristics of sand mining behavior that during the sand mining process, ships may remain still for a long time at one location and then move to another location to remain still for a long time to continue sand mining. In the embodiment of the present application, it is possible to detect whether the ship has a secondary static behavior. The secondary static behavior refers to that when the ship has been static for a long time, the ship has a state conversion trend, that is, it is converted from a moving state to a static state, wherein the static duration corresponding to the long-term static behavior is greater than the static duration threshold. Based on this, when the state conversion of the ship is detected to be static for a second time, the static duration of the ship after switching to the static state can be determined. The determination method of the static duration can refer to the above embodiment, which will not be repeated here. In one example, the ship has been stationary at position A from 10:30 to 12:00 above. Assuming that the static duration threshold is set to 1 hour, the static duration of the ship has exceeded 1 hour, and an early warning mark can be set for the ship. Detecting whether the vessel is stationary for a second time after being stationary for a long time is to detect whether the vessel switches from a moving state to a stationary state after moving. In the above example, the vessel starts to move at 12:00 and stops at position B at 12:25. It is determined that the vessel is still stationary at position B until 13:25, and the second stationary time has reached the stationary time threshold of 1 hour. Therefore, it is determined that the vessel has a corresponding warning mark, and the stationary time is 1 hour. Based on this, it can be determined that the vessel has sand mining behavior.

[0109] Of course, in other embodiments, it is also possible to detect that the vessel switches from a moving state to a stationary state multiple times to determine that the vessel has engaged in sand mining. The specific number of state transitions can be set as needed. The more times the vessel switches state, the greater the possibility of sand mining. This application does not limit the number of state transitions required to determine that the vessel has engaged in sand mining.

[0110] Furthermore, the embodiments of the present application can also use the thermal imaging image of the vessel to determine whether the vessel is engaged in sand mining. During the research on sand mining vessels, the applicant found that sand mining vessels can be camouflaged in appearance, but sand mining requires the use of sand mining units, and the operation of sand mining units requires the release of a large amount of heat energy. Therefore, with the help of the thermal imaging image of the vessel, it is possible to determine whether there is a location with a large amount of heat dissipation in the vessel, and then determine whether there is an operating sand mining unit in the vessel. Specifically, in one embodiment of the present application, the image sequence is a thermal imaging image sequence;

[0111] The transportation vehicle includes a ship, and the abnormal behavior includes sand mining behavior; after determining whether the transportation vehicle has abnormal behavior associated with the area to be detected according to the motion state information, the method further includes:

[0112] S601: In response to determining that the vessel has the sand mining behavior, determining a target thermal imaging image based on the thermal imaging image to be detected collected for the vessel; the area ratio of the vessel in the target thermal imaging image is greater than an area threshold;

[0113] S603: Detecting whether there is a heat source of a sand mining unit in the target thermal imaging image;

[0114] S605: In response to detecting the heat source of the sand mining unit in the target thermal imaging image, determining that the vessel has engaged in illegal sand mining.

[0115] In an embodiment of the present application, in response to determining that the vessel has engaged in the sand mining behavior, a thermal imaging image to be detected collected for the vessel is acquired. The thermal imaging image to be detected can be a thermal imaging image of the vessel collected in real time using a thermal imaging camera after determining a specific vessel. The target thermal imaging image may include a vessel image area detected from the thermal imaging image, and the area proportion of the vessel in the target thermal imaging image is greater than an area threshold. For example, the proportion of the vessel area image in the target thermal imaging image is greater than 80%, 90%, etc. For example, the vessel image area can be detected from the thermal imaging image by the vessel detection model. By detecting whether there is a heat source of a sand mining unit in the target thermal imaging image, it is determined that the vessel has engaged in illegal sand mining.

[0116] In one embodiment of the present application, before determining the target thermal imaging image based on the thermal imaging image collected for the ship, the method may further include:

[0117] The image sequence includes a plurality of continuous visible light images; a visible light image for determining the sand mining behavior is determined from the plurality of visible light images, and a thermal imaging image to be detected collected for the vessel at the time of collecting the determined visible light image; or

[0118] The image sequence includes a plurality of continuous thermal imaging images; the thermal imaging image used to determine the sand mining behavior among the plurality of thermal imaging images is used as the thermal imaging image to be detected.

[0119] In the embodiment of the present application, the thermal imaging image can be obtained in a variety of ways. In one embodiment of the present application, a plurality of visible light images can be used to perform a first sand mining behavior detection, and the thermal imaging image to be detected can be used to perform a second sand mining behavior detection. Specifically, Figure 5As shown, in step 701, a plurality of visible light images can be used to perform the first sand mining behavior detection on the area to be detected. The first sand mining behavior detection is to determine whether the ship has abnormal behavior according to the motion state information of the ship in the visible light image. The specific detection method can be the above embodiment, which is not repeated here. In step 702, a visible light image for determining the sand mining behavior can be determined from the plurality of visible light images. Among them, the visible light image for determining the sand mining behavior can include an image corresponding to the moment when it is determined that there is a ship in the area to be detected that has sand mining behavior. For example, there are 20 visible light images in total, and when the 15th visible light image is detected, it is determined that there is a ship in the area to be detected that has sand mining behavior. In step 703, after determining the visible light image for determining the sand mining behavior, a thermal imaging image to be detected can be collected for the ship at the time of collecting the determined visible light image. For example, in the above example, the time of collecting the 15th visible light image can be obtained, and the thermal imaging image to be detected collected for the ship at the time of collecting can be obtained. The acquisition method may be, for example, to acquire the thermal imaging image to be detected in real time at the acquisition time. In step 704, after acquiring the thermal imaging image to be detected, the thermal imaging image to be detected may be used to perform a second sand mining behavior detection.

[0120] In another embodiment of the present application, the first sand mining behavior detection and the second sand mining behavior detection can also be performed based on the thermal imaging image. Figure 6 As shown, when the image sequence contains multiple continuous thermal imaging images, the image of the area to be detected is collected by an infrared camera device throughout the process. In step 801, multiple thermal imaging images can be used to perform a first sand mining behavior detection on the area to be detected. The first sand mining behavior detection is to determine whether the ship has abnormal behavior based on the motion state information of the ship in the visible light image. The specific detection method can be the above-mentioned embodiment, which will not be repeated here. In step 802, a thermal imaging image for determining the sand mining behavior in the thermal imaging images can be determined, and the thermal imaging image can be used as the thermal imaging image to be detected, wherein the thermal imaging image for determining the sand mining behavior can include an image corresponding to the moment when it is determined that there is a ship in the area to be detected that has sand mining behavior. In step 803, after acquiring the thermal imaging image to be detected, the thermal imaging image to be detected can be used to perform a second sand mining behavior detection.

[0121] In one embodiment of the present application, determining the target thermal imaging image based on the thermal imaging image to be detected collected from the ship may include:

[0122] An image region including the vessel is acquired from the thermal imaging image to be detected and determined as the target thermal imaging image; the target thermal imaging image is a sub-image of the thermal imaging image to be detected.

[0123] In the embodiment of the present application, after determining the thermal imaging image to be detected, an image region containing a ship in the thermal imaging image to be detected can be determined, and the image region can be obtained by, for example, detecting a ship detection model. The image region containing the ship is determined as the target thermal imaging image, that is, the target thermal imaging image is a sub-image of the thermal imaging image to be detected at the determined location.

[0124] Through the above embodiments, for the ship detected to have the sand mining behavior, the sand mining unit is further detected, thereby further improving the accuracy of the sand mining behavior detection.

[0125] In one embodiment of the present application, in response to the fact that the heat source of the sand mining unit is not detected in the target thermal imaging image, it is determined that the ship does not engage in illegal sand mining. That is, even if the ship is detected to be engaged in sand mining, if the sand mining unit is not detected in the ship, it can be determined that the ship does not engage in illegal sand mining.

[0126] In one embodiment of the present application, a neural network may be used to detect whether there is a sand mining machine group image in the target thermal imaging image. Specifically, the detection of whether there is a sand mining machine group heat source in the target thermal imaging image may include:

[0127] The target thermal imaging image is detected using the trained sand mining unit detection network, and whether there is a sand mining unit heat source in the target thermal imaging image is determined based on the detection result; wherein:

[0128] The sand mining machine group detection network is obtained by training based on the thermal imaging image samples that are labeled for the sand mining machine group.

[0129] In the embodiment of the present application, the sand mining group detection network can be trained by deep learning. In a specific embodiment, the sand mining group detection network can be trained in the following manner: first, a plurality of thermal imaging image samples can be obtained, wherein the thermal imaging image samples are marked with the position results of the heat source of the sand mining group, for example, the image area corresponding to the heat source of the sand mining group is framed by a bounding box in the thermal imaging image samples. Then, the thermal imaging image samples are input into the sand mining group detection network, and the sand mining group detection network outputs the prediction results for the sand mining group in the thermal imaging image samples. Based on the difference between the prediction results and the position results of the heat source of the sand mining group marked in the thermal imaging image samples, the network parameters of the sand mining group detection network are iteratively adjusted until the sand mining group detection network meets the preset requirements. Wherein, the preset requirements may include, for example, that the difference is less than or equal to a preset threshold, and may also include that the number of iterative adjustments is greater than or equal to a preset number of thresholds, etc., and the present application is not limited here. It should be noted that the sand mining machine detection network can be trained using a variety of target detection network frameworks. Typically, the target detection network frameworks include RCNN series, SSD, YOLO series, etc., and this application does not limit this.

[0130] In an embodiment of the present application, a deep learning model is used to identify whether there is a heat source of a sand mining unit in the target thermal imaging image. Since the deep learning model is learned based on a large number of samples, this identification method has higher detection accuracy and efficiency.

[0131] Of course, in other embodiments, it is also possible to detect whether there is a heat source of a sand mining group in the target thermal imaging image by performing image analysis on the target thermal imaging image. Specifically, in one embodiment of the present application, the detecting whether there is a heat source of a sand mining group in the target thermal imaging image includes:

[0132] S901: performing binarization processing on the target thermal imaging image to obtain a binary thermal imaging image;

[0133] S903: If it is determined that a connected domain that meets preset conditions exists at a preset position in the binary thermal imaging image, it is determined that a heat source of the sand mining unit is detected in the target thermal imaging image.

[0134] In an embodiment of the present application, the target thermal imaging may first be binarized. Specifically, through the binarization process, the pixel temperature value of the target thermal imaging may be set to two values, one of which represents a high temperature region and the other represents a low temperature region, and the two values ​​may be, for example, 0 and 1. The specific binarization process may determine whether the pixel temperature value of the target thermal imaging image is 0 or 1 according to a preset temperature threshold. In response to the pixel temperature value of the target thermal imaging image being greater than the preset temperature threshold, the pixel temperature value is set to 1; in response to the pixel temperature value of the target thermal imaging image being not greater than the preset temperature threshold, the pixel temperature value is set to 0. The preset temperature threshold may be set to 70°C, 100°C, etc., and the present application does not limit this.

[0135] In an embodiment of the present application, after determining the binary thermal imaging image, the binary thermal imaging image may be subjected to denoising and other processing, for example, an erosion and dilation operation may be performed on the binary thermal imaging image to determine the noise points in the binary thermal imaging image, so as to further improve the accuracy of detection.

[0136] In the embodiment of the present application, the heat source of the sand mining unit is shown as a connected image area, and the pixel temperature value of the connected image area is 1, that is, a high temperature area. Based on this, the connected area analysis can be performed on the binary thermal imaging image, and the connected domain that meets the preset requirements in the binary thermal imaging image can be determined by using the Two-Pass, Seed-Filling seed filling method, etc. The preset requirements may include that the connected domain is a high temperature area. The present application does not limit the method for determining the connected area.

[0137] Of course, after determining the connected domain that meets the preset requirements, it is also possible to determine whether the connected domain is a heat source of a sand mining unit based on the position. In an embodiment of the present application, if it is determined that the connected domain is located at a preset position in the binary thermal imaging image, it is determined that the heat source of the sand mining unit is detected in the target thermal imaging image. Generally, the heat source of the sand mining unit is located at a preset position in the vessel. Based on this, the preset position can include the 30%-70% position from left to right in the target thermal imaging image, and the 0-60% position in height from top to bottom.

[0138] In an embodiment of the present application, when it is determined that the vessel has violated the regulations on sand mining, an alarm message can be generated and sent to the client of the relevant personnel. The alarm message can include information such as the location of the sand mining vessel, the duration of sand mining, and sand mining evidence (such as thermal imaging images of the vessel). Furthermore, the identity of the vessel can even be identified, for example, the appearance characteristics of the vessel can be compared with those of the registered vessels to determine the identity of the vessel and the person in charge, so as to quickly locate the illegal personnel.

[0139] The abnormal behavior detection method for vehicles provided in various embodiments of the present application can track the vehicles in the area to be detected based on the image sequence of the area to be detected, determine the position information of the vehicles in multiple consecutive image frames in the image sequence, and determine the motion state information of the vehicles based on the position information. According to the motion state information, it can be determined whether the vehicle has abnormal behavior associated with the area to be detected. The motion state information belongs to the attributes of the vehicle during the movement process. Therefore, determining whether the vehicle has abnormal behavior based on the motion state information not only has high accuracy and detection efficiency, but also has the advantages of low cost and 24-hour all-weather detection, which meets the needs of abnormal behavior detection of vehicles.

[0140] On the other hand, the present application also provides an abnormal behavior detection device for a vehicle, such as Figure 7 As shown, the abnormal behavior detection device 500 for a vehicle may include:

[0141] An image acquisition module 501 is used to acquire an image sequence of the area to be detected, wherein the image sequence includes a plurality of continuous image frames;

[0142] The target tracking module 503 is used to track the vehicle in the image sequence and determine the position information of the vehicle in the image sequence in the plurality of consecutive image frames;

[0143] A motion feature determination module 505, configured to determine motion state information of the vehicle according to position information of the vehicle in the plurality of consecutive image frames;

[0144] The sand mining behavior determination module 507 is used to determine whether the vehicle has abnormal behavior associated with the area to be detected based on the motion state information.

[0145] The embodiment of the present application also provides a camera device, including a lens group, an image sensor, a memory and a processor, characterized in that a computer program is stored in the memory, and the processor is configured to run the computer program to execute the method described in any of the above embodiments. The lens group may include multiple lenses (convex lenses or concave lenses) for collecting light signals reflected by the table and objects on the table, and transmitting the collected light signals to the image sensor. The image sensor generates an image sequence of the area to be detected based on the light signal.

[0146] The embodiment of the present application further provides a processing device 700, which can be a physical device or a cluster of physical devices, or a virtualized cloud device, such as at least one cloud computing device in a cloud computing cluster. For ease of understanding, the present application uses the processing device 700 as an independent physical device to illustrate the structure of the processing device 700.

[0147] like Figure 8 As shown, the processing device 700 includes: a processor and a memory for storing processor executable instructions; wherein the processor is configured to implement the above-mentioned apparatus when executing the instructions. The processing device 700 includes a memory 701, a processor 703, a bus 705 and a communication interface 707. The memory 701, the processor 703 and the communication interface 707 communicate through the bus 701. The bus 705 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 The bus is represented by only one thick line, but it does not mean that there is only one bus or one type of bus. The communication interface 707 is used for communicating with the outside.

[0148] The processor 703 may be a central processing unit (CPU). The memory 701 may include a volatile memory, such as a random access memory (RAM). The memory 701 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a HDD or a SSD.

[0149] The memory 701 stores executable codes, and the processor 703 executes the executable codes to perform the aforementioned test scenario construction method.

[0150] An embodiment of the present application provides a non-volatile computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the above method is implemented.

[0151] An embodiment of the present application provides a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0152] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing.

[0153] The computer-readable program instructions or codes described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0154] The computer program instructions for performing the operation of the present application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, by utilizing the state information of computer-readable program instructions to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA) or a programmable logic array (PLA), the electronic circuit can execute the computer-readable program instructions to implement various aspects of the present application.

[0155] In some embodiments, the disclosed methods may be implemented as computer program instructions encoded in a machine-readable format on a computer-readable storage medium or on other non-transitory media or articles of manufacture. Fig. 9 Schematically illustrates a conceptual partial view of an example computer program product arranged according to at least some embodiments presented herein, the example computer program product comprising a computer program for executing a computer process on a computing device. In one embodiment, the example computer program product 800 is provided using a signal bearing medium 801. The signal bearing medium 801 may include one or more program instructions 802, which when executed by one or more processors may provide the above-described Figure 1 The functions or parts of the functions described. In addition, Fig. 9 Program instructions 802 in also describe example instructions.

[0156] In some examples, the signal bearing medium 801 may include a computer readable medium 803, such as, but not limited to, a hard drive, a compact disk (CD), a digital video disk (DVD), a digital tape, a memory, a read-only memory (ROM) or a random access memory (RAM), etc. In some embodiments, the signal bearing medium 801 may include a computer recordable medium 804, such as, but not limited to, a memory, a read / write (R / W) CD, a R / W DVD, etc. In some embodiments, the signal bearing medium 801 may include a communication medium 805, such as, but not limited to, a digital and / or analog communication medium (e.g., a fiber optic cable, a waveguide, a wired communication link, a wireless communication link, etc.). Thus, for example, the signal bearing medium 801 may be communicated by a wireless form of the communication medium 805 (e.g., a wireless communication medium that complies with the IEEE 802.11 standard or other transmission protocol). One or more program instructions 802 may be, for example, computer executable instructions or logic implementation instructions. In some examples, a computing device such as the computing device described may be configured to provide various operations, functions, or actions in response to program instructions 802 communicated to the computing device by one or more of a computer-readable medium 803, a computer-recordable medium 804, and / or a communication medium 805. It should be understood that the arrangement described herein is only for the purpose of example. Thus, it will be appreciated by those skilled in the art that other arrangements and other elements (e.g., machines, interfaces, functions, sequences, and functional groups, etc.) can be used instead, and some elements can be omitted together according to the desired results. In addition, many of the elements described are functional entities that can be implemented as discrete or distributed components, or in any appropriate combination and position in conjunction with other components.

[0157] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the device, system, method and computer program product according to multiple embodiments of the present application. In this regard, each square frame in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the functions marked in the square frame can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square frames can actually be executed substantially in parallel, and they can also be executed in reverse order sometimes, depending on the functions involved.

[0158] It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented by hardware (such as a circuit or ASIC (Application Specific Integrated Circuit)) that performs the corresponding function or action, or can be implemented by a combination of hardware and software, such as firmware.

[0159] Although the present invention is described herein in conjunction with various embodiments, in the process of implementing the claimed invention, those skilled in the art may understand and implement other variations of the disclosed embodiments by viewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "one" or "an" does not exclude multiple situations. A single processor or other unit may implement several functions listed in the claims. Certain measures are recorded in different dependent claims, but this does not mean that these measures cannot be combined to produce good results.

[0160] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for detecting abnormal behavior of a vehicle, characterized in that: include: Acquire an image sequence of the area to be detected, wherein the image sequence includes a plurality of continuous image frames; Tracking a vehicle in the image sequence to determine position information of the vehicle in the image sequence in the plurality of consecutive image frames; the vehicle includes a ship; Determining motion state information of the vehicle according to position information of the vehicle in the plurality of consecutive image frames; Determining, based on the motion state information, whether the vehicle has abnormal behavior associated with the area to be detected; the abnormal behavior includes sand mining behavior; The determining, based on the motion state information, whether the vehicle has abnormal behavior associated with the area to be detected includes: In response to the motion state information including at least two sub-motion states, determining a state transition trend of the at least two sub-motion states; the at least two sub-motion states including a stationary state and a moving state; If it is detected that the ship in the image sequence switches from a moving state to a stationary state, then in response to a time length for which the ship is stationary being greater than a stationary time length threshold, it is determined that the ship has abnormal behavior associated with the area to be detected.

2. The method according to claim 1, characterized in that After determining, according to the motion state information, whether the vehicle has abnormal behavior associated with the area to be detected, the method further includes: In response to determining that the vessel has engaged in the sand mining behavior, a target thermal imaging image is determined based on the thermal imaging image to be detected collected for the vessel; the area proportion of the vessel in the target thermal imaging image is greater than an area threshold; Detecting whether there is a heat source of a sand mining unit in the target thermal imaging image; In response to detecting the heat source of the sand mining unit in the target thermal imaging image, it is determined that the vessel has engaged in illegal sand mining.

3. The method according to claim 2, characterized in that After detecting whether there is a heat source of the sand mining unit in the target thermal imaging image, the method further includes: In response to the fact that no heat source of the sand mining unit is detected in the target thermal imaging image, it is determined that the vessel has not engaged in any illegal sand mining activities.

4. The method according to claim 2, characterized in that: Before determining the target thermal imaging image based on the to-be-detected thermal imaging image collected from the vessel, the method further includes: The image sequence includes a plurality of continuous visible light images; a visible light image for determining the sand mining behavior is determined from the plurality of visible light images, and a thermal imaging image collected for the ship at the time of collecting the determined visible light image is obtained as the thermal imaging image to be detected; or The image sequence includes a plurality of continuous thermal imaging images; the thermal imaging image used to determine the sand mining behavior among the plurality of thermal imaging images is used as the thermal imaging image to be detected.

5. The method according to any one of claims 2 to 4, characterized in that: The step of determining a target thermal imaging image based on the thermal imaging image to be detected collected from the vessel comprises: An image area including the vessel is acquired from the thermal imaging image to be detected and determined as the target thermal imaging image; the target thermal imaging image is a sub-image of the thermal imaging image to be detected.

6. The method according to claim 2, characterized in that The detecting whether there is a heat source of a sand mining unit in the target thermal imaging image comprises: The target thermal imaging image is detected using the trained sand mining unit detection network, and whether there is a sand mining unit heat source in the target thermal imaging image is determined based on the detection result; wherein: The sand mining machine group detection network is obtained by training based on the thermal imaging image samples that are labeled for the sand mining machine group.

7. The method according to claim 2, characterized in that The detecting whether there is a heat source of a sand mining unit in the target thermal imaging image comprises: Binarizing the target thermal imaging image to obtain a temperature binary image; If it is determined that a connected domain that meets preset conditions exists at a preset position in the temperature binary image, it is determined that the heat source of the sand mining unit is detected in the target thermal imaging image.

8. The method according to claim 1, characterized in that The target tracking of the vehicle in the image sequence and determining the position information of the vehicle in the image sequence in the plurality of continuous image frames includes: Inputting the plurality of continuous image frames into a vehicle detection model respectively, and outputting the position information and the unique identification of the vehicle in the image frames through the vehicle detection model; According to the unique identification of the vehicle, position information of the vehicle in the multiple consecutive image frames is determined.

9. An abnormal behavior detection device for a vehicle, characterized in that: include: An image acquisition module, used to acquire an image sequence of the area to be detected, wherein the image sequence includes a plurality of continuous image frames; A target tracking module, used to track the vehicle in the image sequence and determine the position information of the vehicle in the image sequence in the plurality of consecutive image frames; Said means of transport include vessels; a motion state determination module, configured to determine the motion state information of the vehicle according to the position information of the vehicle in the plurality of consecutive image frames; An abnormal behavior determination module is used to determine whether the vehicle has abnormal behavior associated with the area to be detected according to the motion state information; the abnormal behavior includes sand mining behavior; The abnormal behavior determination module is specifically used to: in response to the motion state information including at least two sub-motion states, determine the state conversion trend of the at least two sub-motion states; the at least two sub-motion states include a static state and a moving state; if it is detected that the ship in the image sequence is converted from a moving state to a static state, in response to the time length of the ship being in the static state being greater than a static time length threshold, determine that the ship has abnormal behavior associated with the area to be detected.

10. A camera device, comprising a lens group, an image sensor, a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 8.

11. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

12. A computer program product, characterized in that The method comprises a computer-readable code or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the method described in any one of claims 1 to 8.

Images