Object monitoring method, electronic device, and computer-readable storage medium

Abstract

The application discloses an object monitoring method, an electronic device and a computer readable storage medium. The object monitoring method comprises: performing target tracking processing on a first video stream collected for a target region; in response to a suspected object in the first video stream meeting an alarm condition associated with the target region, controlling a first shooting device to track and shoot the suspected object to obtain a second video stream; performing target identification on the second video stream; and in response to the suspected object in the second video stream being a target object associated with the target region, performing alarm, wherein the first video stream and the second video stream are video streams collected by different imaging technologies. The method of the application can improve the accuracy of object monitoring.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an object monitoring method, an electronic device and a computer readable storage medium. BACKGROUND

[0002] With the continuous improvement of economic level and people's living standard, target recognition technology has a wide range of applications in video monitoring, robots, intelligent transportation and other fields.

[0003] The present application found that the accuracy of the current target recognition in motion needs to be further improved. SUMMARY

[0004] The present application provides an object monitoring method, an electronic device and a computer readable storage medium, which can improve the accuracy of object monitoring.

[0005] The first aspect of the present application provides an object monitoring method, which comprises: performing target tracking processing on a first video stream collected for a target area; in response to the presence of a suspected object in the first video stream that meets an alarm condition associated with the target area, controlling a first shooting device to track and shoot the suspected object to obtain a second video stream; performing target recognition on the second video stream; and in response to the suspected object in the second video stream being a target object associated with the target area, performing an alarm, wherein the first video stream and the second video stream are video streams collected by different imaging technologies.

[0006] The second aspect of the present application provides an electronic device, which comprises a processor, a memory and a communication circuit, the processor is coupled to the memory and the communication circuit respectively, the memory stores program data, and the processor executes the program data in the memory to realize the steps in the above method.

[0007] The third aspect of the present application provides a computer readable storage medium, which stores a computer program, and the computer program can be executed by a processor to realize the steps in the above method.

[0008] The present application has the following beneficial effects: since the first video stream and the second video stream are video streams collected by different imaging technologies, when the first video stream and the second video stream are respectively subjected to target detection, target detection is performed according to different characteristics, suspected objects are initially screened out according to the characteristics in the first video stream, and it is determined whether the suspected objects are target objects according to the characteristics in the second video stream, that is, target objects are screened out by combining the characteristics of the two aspects, so that the accuracy of monitoring can be improved regardless of whether the object is in a stationary state or a motion state. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0010] Figure 1 This is a flowchart illustrating one implementation method of the object monitoring method of this application;

[0011] Figure 2 yes Figure 1 A schematic diagram of the first video stream in China;

[0012] Figure 3 yes Figure 1 Another part of the implementation method is illustrated in the flowchart;

[0013] Figure 4 yes Figure 3 A flowchart illustrating step S230;

[0014] Figure 5 This is a schematic diagram of the structure of one embodiment of the electronic device of this application;

[0015] Figure 6 This is a schematic diagram of another embodiment of the electronic device of this application;

[0016] Figure 7 This is a schematic diagram of one embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0018] See Figure 1 , Figure 1 This is a flowchart illustrating one embodiment of the object monitoring method of this application. The object monitoring method includes:

[0019] S110: Perform target tracking processing on the first video stream acquired for the target area.

[0020] The object monitoring method of this application can be executed by any device with algorithm processing capabilities, such as a computer or a mobile phone.

[0021] Wherein, in the target tracking processing of the first video stream, the object in the first video stream is identified first, and then the identified object is tracked to generate a unique identifier corresponding to the object.

[0022] In the target tracking processing of the first video stream, the object in the first video stream is identified first, and then the identified object is tracked to generate a unique identifier corresponding to the object.

[0023] Wherein, in the target tracking processing of the first video stream, in order to improve the accuracy and speed of detection, the first target detection network can be used for detection, and the specific process is as follows: the video frame in the first video stream is input into the first target detection network to obtain the position of the object in the video frame and the corresponding confidence.

[0024] Wherein, the first target detection network is pre-trained, and the training process can include: obtaining rich sample images, manually labeling the rectangular frame of the object position in the sample image to form labeled data, and then inputting the sample image and the labeled data into the first target detection network for network training.

[0025] In an application scenario, the first target detection network can use a lightweight network moblienetv3 suitable for embedded device platform as a basic network to extract features, and connect the detection neck of yolov5 after downsampling 8, 16 and 32 branches, thereby constructing the first target detection network.

[0026] S120: In response to the existence of a suspected object in the first video stream that meets the alarm condition of the target region association, the first shooting device is controlled to track and shoot the suspected object to obtain a second video stream.

[0027] In the target tracking processing of the first video stream, if the object meets the alarm condition, it is preliminarily determined that the object may be a target object, and in order to further determine whether the object is a target object, the first shooting device is controlled to track and shoot the object.

[0028] Wherein, in the picture collected by the first shooting device, the suspected object is at or approximately at the center of the picture and occupies most of the picture, so that the picture collected by the first shooting device can include rich detail features of the suspected object, providing favorable conditions for subsequent judgment of whether it is a target object.

[0029] In this embodiment, the first shooting device is a ball machine, which integrates a camera and a gimbal and can shoot while moving, so that the suspected object can be tracked and shot.

[0030] In the process of controlling the first shooting device to track and shoot the suspected object, a single target tracking algorithm can be used to track the suspected object, and the single target tracking algorithm can use a siamrpn network for feature extraction.

[0031] S130: performing target identification on the second video stream.

[0032] The purpose of performing target identification on the second video stream is to determine whether the suspected object is a target object or a non-target object, i.e., to classify the suspected object.

[0033] Any one frame of video in the second video stream can be identified, or multiple frames of video in the second video stream can be identified, and then a final identification result can be determined based on the multiple identification results, for example, 10 frames of video in the second video stream are identified, corresponding to the suspected object, 10 confidence values are obtained (the higher the confidence value, the greater the possibility of being a target object), and then the 10 confidence values are calculated, such as taking the average, taking the maximum value, taking the median, etc., to obtain a final confidence value, and based on the final confidence value, it is determined whether the suspected object is a target object.

[0034] As can be seen from the above analysis, the suspected object is at or approximately at the center of the picture in the second video stream, so in one application scenario, step S130 only identifies the object at the center of the picture as a target object.

[0035] Wherein due to the accuracy of tracking and shooting, multiple objects may exist in the second video stream at the same time, or the suspected object is not placed in the center of the picture, so in another application scenario, step S130 identifies all objects in the second video stream as target objects, i.e., all objects in the picture are assumed to be suspected objects.

[0036] In order to improve the accuracy of target identification, a second target detection network is used to identify the target in the second video stream.

[0037] In order to ensure the accuracy of classifying the suspected object, the second target detection network is based on a dcl fine-grained classification algorithm for classification, compared to a general algorithm, the second target detection network pays more attention to local features in the training stage, and introduces an adversarial network branch, which can make the classification result more accurate.

[0038] S140: in response to the suspected object in the second video stream being a target object associated with a target region, an alarm is performed, wherein the first video stream and the second video stream are video streams collected by different imaging technologies.

[0039] When it is determined that the suspected object is a target object, an alarm is performed.

[0040] In the embodiment, the first video stream and the second video stream are video streams collected by different imaging technologies, that is, the specific features presented by the pictures in the first video stream are different from those in the second video stream.

[0041] That is, when target detection is performed on the first video stream and the second video stream respectively, the detection is performed according to different features, that is, suspected objects are initially screened out according to the features in the first video stream, and whether the suspected objects are target objects is determined according to the features in the second video stream, that is, the target objects are screened out by combining the features of the two aspects, so that the monitoring accuracy can be improved regardless of whether the objects are in a motion state or a static state.

[0042] In the embodiment, the first video stream is a video stream collected by a thermal imaging technology, and the second video stream is a video stream collected by a visible light imaging technology. That is, the first photographing device is a visible light camera, and the second photographing device is a thermal imaging camera.

[0043] In the embodiment, the first video stream is a video stream collected by a thermal imaging technology, and the second video stream is a video stream collected by a visible light imaging technology. That is, the first photographing device is a visible light camera, and the second photographing device is a thermal imaging camera.

[0044] Since the thermal imaging camera has strong anti-interference capability, the clarity of the collected image will not be affected by environmental factors such as heavy fog and rainy weather, so setting the second photographing device as a thermal imaging camera can ensure that each object entering the target area can be identified.

[0045] The picture collected by the visible light camera is clear and can include rich detailed features, so setting the first photographing device as a visible light camera can further improve the monitoring accuracy.

[0046] In an embodiment, the first photographing device and the second photographing device are integrated in the same housing, that is, a binocular camera is used for photographing, the binocular camera including a thermal imaging lens (that is, the second photographing device) and a visible light lens (that is, the first photographing device).

[0047] In the embodiment, the second photographing device is a gun camera, and the second photographing device remains stationary when collecting the first video stream.

[0048] In another embodiment, the first photographing device and the second photographing device are two independent devices.

[0049] In the embodiment, the first photographing device and the second photographing device are two independent devices.

[0050] In the embodiment, the second shooting device has a larger shooting field of view than the first shooting device, specifically, the second shooting device performs global monitoring on the target region, and when it is determined that there is a suspected object, the first shooting device performs tracking shooting on the suspected object, i.e., local monitoring, that is, the proportion of the suspected object in the picture of the first video stream is smaller than the proportion of the suspected object in the picture of the second video stream.

[0051] Of course, in other embodiments, the shooting field of view of the second shooting device can also be equal to the shooting field of view of the first shooting device, that is, at this time, the proportion of the suspected object in the picture of the first video stream is equal to the proportion of the suspected object in the picture of the second video stream.

[0052] In an application scenario, the target object includes a vehicle, and the target region includes a no-entry region where vehicles are prohibited from entering; and the alarm condition includes that the suspected object enters the no-entry region.

[0053] The vehicle can be any type of vehicle, such as a motor vehicle, a bicycle, or a ship.

[0054] In order to facilitate understanding of the scheme of the application scenario, the scheme is described in combination with Figure 2 the figure in Figure 2 is an image in the first video stream collected for the target region, which includes the no-entry region 10, and when the suspected object 20 enters the no-entry region 10, it is determined that the suspected object 20 meets the alarm condition.

[0055] The subsequent step S130 performs target identification on the suspected object, and when it is determined that the suspected object is a vehicle, an alarm is performed.

[0056] In other application scenarios, the subsequent step S130 can also be that target identification is performed on the suspected object, and an alarm is performed only when it is determined that the suspected object is a specific type of vehicle, such as a fishing boat.

[0057] That is, in this application scenario, when the first video stream is processed by the target tracking process in step S110, any type of object in the first video stream is tracked, and then the object that enters the no-entry region is determined as the suspected object, and then the first shooting device is controlled to perform tracking shooting on the suspected object to obtain the second video stream, and then the second video stream is processed by target identification, and if it is determined that the suspected object is a vehicle or a specific type of vehicle, such as a fishing boat, an alarm is performed.

[0058] In order to improve the accuracy of detection and avoid false positives, the standard for determining that the suspected object enters the no-entry area can be: in response to a continuous preset number of video frames in the first video stream being target video frames, it is determined that the suspected object enters the no-entry area; wherein in the target video frame, the center point of the detection box of the suspected object is located in the no-entry area, and the confidence of the suspected object exceeds the confidence threshold.

[0059] The preset number and the confidence threshold can be set according to the actual application scenario, for example, the preset number is 10 frames or 25 frames, and the confidence threshold is 0.5 or 0.65.

[0060] In order to better illustrate, the preset number is 10 frames and the confidence threshold is 0.65 to illustrate the above-mentioned standard for determining that the suspected object enters the no-entry area:

[0061] When the center point of the detection box of the suspected object is first detected in the no-entry area, the confidence of the suspected object and the frame number are accumulated. During the accumulation process, if the confidence of 10 consecutive video frames reaches 0.65 or more, and the center point of the detection box of the suspected object is in the no-entry area in the 10 consecutive video frames, it is determined that the suspected object enters the no-entry area.

[0062] It should be noted that in other embodiments, the standard for determining that the suspected object enters the no-entry area can also be: as long as the center point of the detection box of the suspected object is in the no-entry area, it is determined that the suspected object enters the no-entry area. In summary, the application does not limit the standard for determining that the suspected object enters the no-entry area.

[0063] In another application scenario, the suspected object is a ship, and the target area includes a ship channel area. At this time, the alarm condition includes: the suspected object is in a stationary state in the channel area, and the stationary time in the stationary state reaches a time threshold.

[0064] In the channel area, all ships must be in a moving state. The ship can be any type of ship, such as a cargo ship, a sand mining ship, or a fishing boat.

[0065] When the suspected object is in a stationary state in the channel area, and the stationary time in the stationary state reaches a time threshold, it is determined that the suspected object meets the alarm condition. The subsequent step S130 performs target identification on the suspected object. When it is determined that the suspected object is a ship, or a specific type of ship, such as a fishing boat, it is determined to alarm.

[0066] That is to say, in this application scenario, step S110 only tracks the ship in the first video stream when performing target tracking processing on the first video stream, and then determines the ship entering the channel region as a suspected object, and then controls the first shooting device to track and shoot the suspected object to obtain the second video stream, and then performs target identification on the second video stream. Then, the suspected object is indeed a ship, or a specific type of ship, such as a fishing boat, and an alarm is processed.

[0067] It should be noted that in other embodiments, the alarm condition can also only include that the suspected object is in a stationary state in the channel region, that is, as long as the suspected object is in a stationary state in the channel region, it is determined that the suspected object meets the alarm condition without the need for the stationary time to reach the time threshold.

[0068] The time threshold can be set according to the actual scene requirements, for example, the time threshold is 5 minutes or 30 minutes, etc.

[0069] The process of determining whether the suspected object is in a stationary state includes: calculating a detection box of the suspected object corresponding to the current frame, and comparing a percentage of displacement of the detection box of the suspected object corresponding to the previous frame in the x direction ε x and a percentage of displacement of the detection box of the suspected object corresponding to the previous frame in the y direction ε y If both ε x and ε y are less than a threshold α, it is determined that the suspected object is in a stationary state at the current time.

[0070] The calculation formulas of ε x and ε y are as follows:

[0071]

[0072] Wherein, δ x is the displacement of the detection box of the suspected object in the x direction, δ y is the displacement of the detection box of the suspected object in the y direction, x t+1 is the horizontal coordinate of the center point of the detection box of the suspected object corresponding to the current frame, x t is the horizontal coordinate of the center point of the detection box of the suspected object corresponding to the previous frame, y t+1 is the vertical coordinate of the center point of the detection box of the suspected object corresponding to the current frame, y t is the vertical coordinate of the center point of the detection box of the suspected object corresponding to the previous frame, w t+1 is the width of the detection box of the suspected object corresponding to the current frame, w t is the width of the detection box of the suspected object corresponding to the previous frame, h t+1 is the height of the detection box of the suspected object corresponding to the current frame, h tA height of a detection frame corresponding to the suspected object of the previous frame.

[0073] In the process of target tracking by using the tracking algorithm, there is a defect that when the target intersects with another target, the tracking algorithm may break the tracking and change the identification of the target, and as long as the identification of the target is changed, it is considered as two different targets. That is, after the intersection occurs, the identification of the target may be changed. Under this premise, during the process of the suspected object being in a stationary state, if another ship intersects with it before its stationary time reaches the time threshold, the identification of the suspected object may be changed, thereby stopping the accumulation of the stationary time of the suspected object, and further, the phenomenon that the suspected object is always in a stationary state but its stationary time does not reach the time threshold occurs.

[0074] In order to avoid the above phenomenon, referring to Figure 3 The method of the application further comprises:

[0075] S210: When the suspected object is first detected to be in a stationary state in the channel area, start timing to accumulate the stationary time of the suspected object in the stationary state.

[0076] In which, when the suspected object is first detected to be in a stationary state in the channel area, start accumulating its stationary time from zero.

[0077] S220: When the suspected object is in a stationary state, in response to the suspected object intersecting with the first ship, stop timing to obtain the first time.

[0078] During the accumulation of the stationary time, if the suspected object is not detected to intersect with the first ship, continue to accumulate until the time threshold is reached, and determine that the suspected object meets the alarm condition.

[0079] If the suspected object is detected to intersect with the first ship during the accumulation of the stationary time, stop accumulating the stationary time and obtain the current accumulated first time.

[0080] S230: After the preset time arrives, determine whether there is a second ship in the original position of the suspected object.

[0081] If there is, execute step S240, and if there is not, execute step S250.

[0082] After the preset time elapses, it is determined that the first ship intersecting with the suspected object has left, and the two ships are no longer in an intersecting state, and at this time, it is determined whether there is a second ship in the original position of the suspected object.

[0083] S240: Determine the second ship as a suspected object, and start counting the still time of the suspected object in a stationary state from the sum of the first time and the preset time.

[0084] If there is a second ship at the original position of the suspected object, it is determined that the second ship is the suspected object, and the preset time of the intersection is accumulated on the basis of the first time, and the still time of the suspected object is continued to accumulate on the basis of the sum of the first time and the preset time.

[0085] S250: Determine that the suspected object is no longer in a stationary state.

[0086] If there is no second ship at the original position of the suspected object, it can be determined that the suspected object has left and will no longer meet the alarm condition.

[0087] It should be noted that in other embodiments, without considering the accuracy, the case of tracking breakage when the target intersects can also not be considered, at this time, in the process of the suspected object being in a stationary state, no intersection judgment is performed, as long as the identification of the suspected object does not change and the suspected object is in a stationary state, the accumulation of the still time is not stopped, and as long as the identification of the suspected object changes, the suspected object is re-identified as a new ship.

[0088] The standard for determining that the suspected object intersects with the first ship can be that, in response to the intersection-over-union of the area occupied by the detection box of the suspected object and the area occupied by the detection box of the first ship in the same video frame in the first video stream exceeding a first intersection-over-union threshold, it is determined that the suspected object intersects with the first ship.

[0089] The first intersection-over-union threshold can be set according to actual conditions, for example, set to 0.5 or 0.6, etc.

[0090] In the same video frame, if the intersection-over-union of the area occupied by the detection box of the suspected object and the area occupied by the detection box of the first ship exceeds the first intersection-over-union threshold, it is determined that the suspected object intersects with the first ship.

[0091] It should be noted that in other embodiments, the standard for determining that the suspected object intersects with the first ship can also be that, in response to the intersection of the area occupied by the detection box of the suspected object and the area occupied by the detection box of the first ship in the same video frame in the first video stream, it is determined that the suspected object intersects with the first ship. In other words, at this time, as long as the intersection of the two detection boxes occurs, it is determined that the two ships intersect.

[0092] Referring to Figure 4 , the step S230 of determining whether there is a second ship at the original position of the suspected object includes:

[0093] S231: Obtain a first area occupied by the detection frame of the suspected object before intersecting with the first ship.

[0094] The first area occupied by the detection frame of the suspected object refers to an area occupied by the detection frame of the suspected object in a video frame of the first video stream before intersecting with the first ship.

[0095] S232: Determine an intersection-over-union of the second area occupied by the detection frame of each ship in the channel area and the first area.

[0096] After the preset time, the second area occupied by the detection frame of each ship in the channel area is obtained, and then the intersection-over-union of each second area and the first area is determined respectively to obtain the intersection-over-union corresponding to each ship.

[0097] S233: Determine the highest intersection-over-union in the intersection-over-union corresponding to each ship.

[0098] S234: Determine whether the highest intersection-over-union exceeds a second intersection-over-union threshold.

[0099] If the determination result is that it exceeds, step S235 is executed, and if the determination result is that it does not exceed, step S236 is executed.

[0100] The second intersection-over-union threshold can be set according to actual conditions, for example, 0.8, or 0.85, etc.

[0101] S235: Determine that there is a second ship at the original position of the suspected object.

[0102] S236: Determine that there is no second ship at the original position of the suspected object.

[0103] If the highest intersection-over-union exceeds the second intersection-over-union threshold, the ship corresponding to the highest intersection-over-union is determined as the second ship, which will be regarded as the suspected object in the subsequent process and the stationary time will be accumulated continuously. If the highest intersection-over-union does not exceed the second intersection-over-union threshold, it is determined that the suspected object changes from a stationary state to a moving state during the intersection process.

[0104] The preset time in the above step S230 can be a fixed time, for example, 1 minute or 2 minutes, etc. At this time, the preset time is irrelevant to the actual scene and is set by the designer in advance. Alternatively, the preset time can be related to the actual scene, and at this time, the determination process of the preset time includes:

[0105] (c) Obtain the moving speed of the first ship.

[0106] (d) Determine the preset time according to the length of the suspected object and the moving speed.

[0107] The speed from the intersection to the separation of the two ships depends on the moving speed of the ships, while the suspected object is in a static state, so the moving speed of the first ship is obtained, wherein the moving speed of the detection frame of the first ship represents the moving speed of the first ship, so the moving speed v of the first ship can be calculated according to the following formula:

[0108] v = 25 | x' t+1 -x' t |, wherein x' t+1 is the horizontal coordinate of the center point of the detection frame of the first ship corresponding to the current frame, and x' t is the horizontal coordinate of the center point of the detection frame of the first ship corresponding to the previous frame.

[0109] Since the width of the detection frame of the suspected object represents the length of the suspected object, the preset time t can be determined by using the following formula:

[0110] wherein w is the width of the detection frame of the suspected object.

[0111] That is, the embodiment determines that the first ship no longer intersects with the suspected object after the first ship moves twice the length of the suspected object and then moves for another 2 seconds.

[0112] It should be noted that other embodiments can also directly determine that the first ship no longer intersects with the suspected object after the first ship moves twice the length of the suspected object, that is, the preset time t can be calculated according to the following formula at this time:

[0113]

[0114] Alternatively, in other embodiments, the preset time t can also be calculated according to the following formula:

[0115] wherein w1 is the width of the detection frame of the suspected object, and w2 is the width of the detection frame of the first ship.

[0116] In order for the relevant departments to manually confirm, control and alarm the target object, the process of alarming includes:

[0117] (a) determining the position of the suspected object according to the position of the first shooting device and the zoom ratio of the first shooting device for tracking and shooting the suspected object.

[0118] According to the height, longitude and latitude of the first shooting device, and the zoom ratio of the first shooting device for tracking and shooting the suspected object, the position of the suspected object can be determined.

[0119] (b) generating an alarm information, wherein the alarm information carries the corresponding relationship between the suspected object and the position of the suspected object.

[0120] The alarm information can be text information or voice information.

[0121] For example, when the suspected object is a ship and the target object is a fishing boat, the correspondence between the suspected object and the latitude and longitude is stored in the form of a table, such as Table 1 below.

[0122] Table 1: Fishing boat alarm event

[0123]

[0124] In Table 1 above, the generation time is the time when the suspected object triggers the alarm condition.

[0125] As can be seen from Table 1, this embodiment not only saves the correspondence between the ship and its position when the suspected object is a fishing boat, but also saves the correspondence when the suspected object is not a fishing boat. In order to facilitate subsequent evidence collection, the second video stream can also be saved correspondingly.

[0126] Referring to Figure 5 , Figure 5 is a structural schematic diagram of an embodiment of an electronic device of the present application. The electronic device 200 includes a processor 210, a memory 220, and a communication circuit 230, the processor 210 is respectively coupled to the memory 220 and the communication circuit 230, the memory 220 stores program data, and the processor 210 implements the method steps in any of the above embodiments by executing the program data in the memory 220, wherein the detailed steps can be referred to the above embodiments and will not be repeated here.

[0127] Among them, the electronic device 200 can be a computer, a mobile phone or any device with algorithm processing capability, which is not limited here.

[0128] Referring to Figure 6 , Figure 6 is a structural schematic diagram of another embodiment of an electronic device of the present application. The electronic device 300 includes a first detection module 310, a control module 320, a second detection module 330, and an alarm module 340.

[0129] The first detection module 310 is used for target tracking processing on the first video stream collected for the target area.

[0130] The control module 320 is connected with the first detection module 310, and is used for controlling the first shooting device to track and shoot the suspected object when there is a suspected object in the first video stream that meets the alarm condition associated with the target area, to obtain a second video stream.

[0131] The second detection module 330 is connected with the control module 320, and is used for target identification on the second video stream.

[0132] The alarm module 340 is connected with the second detection module 330, and is configured to perform alarm when the suspected object in the second video stream is the target object associated with the target area, wherein the first video stream and the second video stream are video streams collected by different imaging technologies.

[0133] In an embodiment, the first video stream is a video stream collected by a thermal imaging technology, and the second video stream is a video stream collected by a visible light imaging technology.

[0134] In an embodiment, the target object includes a vehicle, and the target area includes a no-entry area in which the vehicle is prohibited from entering; and the alarm condition includes that the suspected object enters the no-entry area.

[0135] In an embodiment, the first detection module 310 is configured to determine that the suspected object enters the no-entry area when a continuous preset number of video frames in the first video stream are target video frames; wherein in the target video frame, a center point of a detection box of the suspected object is located in the no-entry area, and a confidence of the suspected object exceeds a confidence threshold.

[0136] In an embodiment, the suspected object is a ship, and the target area includes a ship channel area; and the alarm condition includes that the suspected object is in a stationary state in the ship channel area, and a stationary time of the suspected object in the stationary state reaches a time threshold.

[0137] In an embodiment, the first detection module 310 is further configured to: start timing to accumulate the stationary time of the suspected object in the stationary state when the suspected object is first detected to be in the stationary state in the ship channel area; stop timing to obtain a first time in response to the suspected object intersecting with a first ship while the suspected object is in the stationary state; after a preset time arrives, determine whether a second ship exists at an original position of the suspected object; if the second ship exists, determine the second ship as the suspected object, and start timing from a sum of the first time and the preset time to continue accumulating the stationary time of the suspected object in the stationary state; and if the second ship does not exist, determine that the suspected object is no longer in the stationary state.

[0138] In an embodiment, the first detection module 310 is further configured to: determine that the suspected object intersects with the first ship in response to an intersection-over-union of an area occupied by a detection box of the suspected object and an area occupied by a detection box of the first ship exceeding a first intersection-over-union threshold in a same video frame in the first video stream.

[0139] In an embodiment, the first detection module 310 is further configured to: acquire a first area occupied by the detection frame of the suspected object before the first ship intersects with the suspected object; determine an intersection-over-union ratio of a second area occupied by the detection frame of each ship in the channel area and the first area; determine a highest intersection-over-union ratio among the intersection-over-union ratios corresponding to each ship; and determine that the second ship at the original position of the suspected object exists in response to the highest intersection-over-union ratio exceeding a second intersection-over-union ratio threshold.

[0140] In an embodiment, the first detection module 310 is further configured to: acquire a moving speed of the first ship; and determine a preset time according to the length of the suspected object and the moving speed.

[0141] In an embodiment, the alarm module 340 is further configured to: determine the position of the suspected object according to the position of the first shooting device and a zoom ratio at which the first shooting device tracks and shoots the suspected object; and generate alarm information carrying a corresponding relationship between the suspected object and the position of the suspected object.

[0142] The electronic device 300 can be a computer, a mobile phone, or any device having an algorithm processing capability, which is not limited herein.

[0143] The electronic device 300 implements the method steps in any of the above embodiments when working, and the detailed steps can be referred to the above embodiments, which are not repeated here.

[0144] Referring to Figure 7 , Figure 7 FIG. 4 is a structural schematic diagram of an embodiment of a computer readable storage medium of the present application. The computer readable storage medium 400 stores a computer program 410, which can be executed by a processor to implement the steps in any of the above methods.

[0145] The computer readable storage medium 400 can be a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or any device that can store the computer program 410, or can be a server that stores the computer program 410 and sends the stored computer program 410 to other devices for running, or can run the stored computer program 410 itself.

[0146] The above description is only an embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation based on the content of the specification and drawings, or direct or indirect application in other related technical fields, is also included in the patent protection scope of the present application.

Claims

1. An object monitoring method, characterized in that, The method includes: Perform target tracking processing on the first video stream acquired for the target area; In response to the presence of a suspected object in the first video stream that meets the alarm conditions associated with the target area, the first shooting device is controlled to track and shoot the suspected object to obtain a second video stream; Target identification is performed on the second video stream; An alarm is triggered in response to the fact that the suspected object in the second video stream is a target object associated with the target area, wherein the first video stream and the second video stream are video streams acquired through different imaging technologies; The suspected object is a vessel, and the target area includes the vessel's shipping lane area; The alarm conditions include: the suspected object is stationary in the waterway area, and the stationary time reaches a time threshold. The method further includes: When the suspected object is first detected to be stationary in the waterway area, a timer is started to accumulate the stationary time of the suspected object. When the suspected object is stationary, the timing stops in response to the suspected object intersecting with the first vessel, and the first time is obtained; After the preset time has elapsed, determine whether there is a second vessel at the original location of the suspected object; If the second vessel exists, the second vessel is identified as the suspected object, and the time is started from the sum of the first time and the preset time to continue accumulating the static time of the suspected object in a static state; If the second vessel does not exist, then the suspected object is determined to no longer be in a stationary state.

2. The method according to claim 1, characterized in that, The first video stream is a video stream acquired using thermal imaging technology, and the second video stream is a video stream acquired using visible light imaging technology.

3. The method according to claim 1, characterized in that, The target objects include vessels, and the target areas include restricted areas where the vessels are prohibited from entering; The alarm conditions also include: the suspected object entering the restricted area.

4. The method according to claim 3, characterized in that, The method further includes: In response to a predetermined number of consecutive video frames in the first video stream being target video frames, it is determined that the suspected object has entered the restricted area; In the target video frame, the center point of the detection box of the suspected object is located in the restricted area, and the confidence level of the suspected object exceeds the confidence level threshold.

5. The method according to claim 1, characterized in that, The method further includes: In response to the fact that, in the same video frame of the first video stream, the intersection-union ratio (IU) of the detection box area of ​​the suspected object and the detection box area of ​​the first vessel exceeds a first IU threshold, it is determined that the suspected object intersects with the first vessel.

6. The method according to claim 1, characterized in that, The step of determining whether a second vessel exists at the original location of the suspected object includes: Obtain the first region occupied by the detection box of the suspected object before intersecting with the first vessel; Determine the intersection-over-union ratio (IoU) between the second region occupied by the detection frames of each vessel in the waterway area and the first region; Among the crossover and union ratios corresponding to each of the aforementioned vessels, the highest crossover and union ratio is determined; In response to the highest crossover ratio exceeding a second crossover ratio threshold, it is determined that the second vessel exists at the original location of the suspected object.

7. The method according to claim 6, characterized in that, Before determining whether a second vessel exists at the original location of the suspected object after the preset time has elapsed, the method further includes: Obtain the speed of the first ship; The preset time is determined based on the length of the suspected object and its moving speed.

8. The method according to claim 1, characterized in that, The steps for triggering the alarm include: The location of the suspected object is determined based on the position of the first shooting device and the variable magnification of the first shooting device in tracking and shooting the suspected object. An alarm message is generated, which carries the correspondence between the suspected object and its location.

9. An electronic device, characterized in that, The electronic device includes a processor, a memory, and a communication circuit. The processor is coupled to the memory and the communication circuit. The memory stores program data. The processor executes the program data in the memory to implement the steps of the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can be executed by a processor to implement the steps of the method as described in any one of claims 1-8.

Images