Night video monitoring method and device, electronic equipment and storage medium

Abstract

This invention provides a nighttime video monitoring method, apparatus, electronic device, and storage medium. The method includes: processing raw nighttime video to obtain image-enhanced nighttime video; determining semantic features based on the image-enhanced nighttime video and determining motion features based on the raw nighttime video; determining a prediction result based on the semantic features and motion features; issuing an early warning based on the prediction result; and the prediction result characterizing whether illegal behavior exists in the nighttime video. The nighttime video monitoring method, apparatus, electronic device, and storage medium provided by this invention accurately identify targets triggering early warnings by extracting semantic features from the image-enhanced nighttime video and motion features from the raw nighttime video, combining the semantic features and motion features. This avoids false alarms, reduces the need for additional auxiliary equipment, and saves construction costs.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and specifically to a nighttime video monitoring method, apparatus, electronic device, and storage medium. Background Technology

[0002] In daily life, insufficient light at night and the deep fatigue of security personnel lead to poor nighttime theft prevention in monitored areas. Nighttime images also suffer from low brightness and insufficient contrast due to insufficient lighting, making them unsuitable for direct analysis and processing.

[0003] Existing nighttime security systems mostly rely on infrared detection technology and sensors installed on doors. The former involves deploying a large number of infrared emitters around the building to form an infrared protective net, while the latter involves installing pressure sensors on the doors. When the door is knocked or pried open, the sensors detect a change in pressure on the door, triggering an alarm.

[0004] However, using infrared devices or sensors to achieve anti-theft protection has several drawbacks. First, the construction cost is very high for large-scale application scenarios. Second, it cannot identify targets that violate the anti-theft alarm, resulting in a high false alarm rate. Summary of the Invention

[0005] This invention provides a nighttime video monitoring method, device, electronic equipment, and storage medium to solve the technical problem of high false alarm rate.

[0006] In a first aspect, the present invention provides a nighttime video monitoring method, comprising:

[0007] The original nighttime video is processed to obtain an image-enhanced nighttime video;

[0008] Semantic features are determined from the image-enhanced nighttime video, and motion features are determined from the original nighttime video;

[0009] The prediction result is determined based on the semantic features and the motion features, and an early warning is issued based on the prediction result, wherein the prediction result indicates whether there is illegal behavior in the night video.

[0010] In one embodiment, processing the original nighttime video to obtain an image-enhanced nighttime video includes:

[0011] The original nighttime video is input into the image enhancement model to obtain the enhanced nighttime video output by the image enhancement model.

[0012] The image enhancement model includes a generation layer and a discrimination layer. The generation layer is used to generate a new night video from the original night video. The discrimination layer is used to determine whether the new night video meets the similarity condition based on the reflectance brightness of the new night video and the reflectance brightness of the standard white-room image, and then outputs the new night video as the image-enhanced night video. The white-room image and any frame of the night video are images captured from the same scene.

[0013] In one embodiment, determining semantic features based on the image-enhanced night video and determining motion features based on the original night video includes:

[0014] The enhanced nighttime video is input into the SlowFast network model to obtain the semantic and motion features output by the SlowFast network model.

[0015] The SlowFast network model includes a low-frequency channel and a high-frequency channel. The low-frequency channel is used to determine semantic features based on the image-enhanced night video, and the high-frequency channel is used to determine motion features based on the original night video.

[0016] In one embodiment, determining the prediction result based on the semantic features and the motion features, and issuing an early warning based on the prediction result, includes:

[0017] If continuous abnormal actions are determined based on the semantic features and motion features, then the image-enhanced nighttime video is saved and a warning message is issued.

[0018] In one embodiment, the method further includes: obtaining facial information of a person making continuous abnormal movements in a nighttime video based on the semantic features and the motion features, and storing the facial information in a preset blacklist.

[0019] In one embodiment, the method further includes the step of determining the location of the warning point, including:

[0020] Determine the image coordinates of the warning point on the nighttime video;

[0021] The image coordinates are converted into the first world coordinates corresponding to the warning point. The first world coordinates are coordinates established with the warning camera as the origin. The night video captured by the warning camera contains the warning point.

[0022] The first world coordinates are converted into the second world coordinates corresponding to the warning point, and the second world coordinates are coordinates established with the monitoring center as the origin;

[0023] Based on the second world coordinates and the world coordinates of multiple preset landmarks, determine multiple relative distances, and then determine the landmark closest to the warning point based on these multiple relative distances.

[0024] In one embodiment, the step of converting the image coordinates into first world coordinates corresponding to the warning point includes:

[0025] The first world coordinates corresponding to the warning point are determined using a first formula based on the image coordinates. The first formula is as follows:

[0026] ;

[0027] in, It is First World coordinates. It is a transformation matrix. It's the focal length. It is the length of an image pixel. It is the width of an image in pixels. These are the coordinates of the early warning camera in the image. These are the coordinates of the warning point in the image. It is the depth value of the warning point in the image. It is a translation matrix.

[0028] In one embodiment, the step of converting the first world coordinates into the second world coordinates corresponding to the warning point includes:

[0029] The second world coordinates corresponding to the warning point are determined using a second formula based on the first world coordinates. The second formula is as follows:

[0030] ;

[0031] in, These are the coordinates of the monitoring center. It is a second-world coordinate system. It is a first-world coordinate system.

[0032] Secondly, the present invention provides a nighttime video monitoring device, comprising:

[0033] The enhancement module is used to process the original night video to obtain an enhanced night video.

[0034] The determination module is used to determine semantic features based on the image-enhanced night video and motion features based on the original night video;

[0035] The early warning module is used to determine the prediction result based on the semantic features and the motion features, and to issue an early warning based on the prediction result, wherein the prediction result indicates whether there is illegal behavior in the night video.

[0036] Thirdly, the present invention provides an electronic device, including a memory and a memory storing a computer program, wherein the processor executes the program to implement the steps of the nighttime video monitoring method described in the first aspect.

[0037] Fourthly, the present invention provides a processor-readable storage medium storing a computer program for causing the processor to perform the steps of the nighttime video monitoring method described in the first aspect.

[0038] The nighttime video monitoring method, device, electronic equipment, and storage medium provided by this invention extract semantic features from the enhanced nighttime video and motion features from the original nighttime video, and combine the semantic features and motion features to accurately identify targets that trigger warnings, avoiding false alarms and reducing the need for additional auxiliary equipment, thus saving construction costs. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0040] Figure 1 This is a flowchart illustrating the nighttime video monitoring method provided by the present invention;

[0041] Figure 2 This is a schematic diagram of the structure of the nighttime video monitoring device provided by the present invention;

[0042] Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0044] The following is combined with Figures 1-3 The present invention describes a nighttime video monitoring method, apparatus, electronic device, and storage medium.

[0045] Figure 1A flowchart illustrating the nighttime video monitoring method provided by this invention is shown below. Figure 1 The nighttime video monitoring method includes the following steps:

[0046] 11. Process the original nighttime video to obtain an image-enhanced nighttime video;

[0047] 12. Determine semantic features from the enhanced nighttime video and motion features from the original nighttime video;

[0048] 13. Determine the prediction results based on semantic and motion features, and issue warnings based on the prediction results. The prediction results indicate whether there are illegal behaviors in nighttime videos.

[0049] Regarding steps 11-13, it should be noted that in this invention, the method is applied to anti-theft scenarios on the periphery of communities or parks. A monitoring center and multiple cameras are set up within the community or park. The original nighttime video is the video captured by each camera at night (under an arrangement with no blind spots).

[0050] The original nighttime video is processed to obtain an image-enhanced nighttime video, which facilitates subsequent monitoring and processing of the nighttime video.

[0051] Semantic features are extracted from image-enhanced nighttime videos. These semantic features can be divided into a visual layer, an object layer, and a conceptual layer. The visual layer is the lowest level of understanding, including features such as color, texture, and shape; these features are collectively referred to as low-level semantic features. The object layer is the intermediate layer, containing attribute features, i.e., the state of an object at a given moment. The conceptual layer is the highest level, representing the information in the image that is closest to human understanding. For example, if an image contains sand, blue sky, and seawater, the visual layer distinguishes these elements, the object layer represents the sand, blue sky, and seawater, and the conceptual layer represents the beach. Therefore, these semantic features can represent various information in each frame of the video.

[0052] Motion features are extracted from raw nighttime videos. These motion features represent the continuous movements of animals (such as humans) appearing in the video. For example, in detecting unusual climbing in community security, climbing is a key action for judgment. Climbing is also a continuous process, and using images alone as the data source for action recognition can lead to significant misjudgments. Therefore, combining motion features improves the accuracy of recognition.

[0053] Then, based on the extracted semantic and motion features, the prediction result is determined, and an early warning is issued based on the prediction result. The prediction result indicates whether there is illegal behavior in the nighttime video. In other words, based on the extracted semantic and motion features, it is possible to determine whether someone has broken into the monitored area from the nighttime video. If someone breaks into the monitored area, an early warning message is issued, and the security personnel in the monitored area are notified.

[0054] The nighttime video monitoring method provided by this invention extracts semantic features from the enhanced nighttime video and motion features from the original nighttime video. By combining the semantic features and motion features, the method accurately identifies targets that trigger warnings, avoids false alarms, reduces the need for additional auxiliary equipment, and saves construction costs.

[0055] In a further invention of the aforementioned nighttime video monitoring method, the process of processing the original nighttime video to obtain image-enhanced nighttime video is explained as follows:

[0056] The original nighttime video is input into the image enhancement model to obtain the enhanced nighttime video output by the image enhancement model.

[0057] The image enhancement model includes a generation layer and a discrimination layer. The generation layer is used to generate a new night video from the original night video. The discrimination layer is used to determine whether the new night video meets the similarity condition based on the reflectance brightness of the new night video and the reflectance brightness of the standard white-room image, and then outputs the new night video as the image-enhanced night video. Any frame in the white-room image and the night video is an image captured from the same scene.

[0058] It should be noted that this image enhancement model uses a GAN network to enhance the original nighttime video. This GAN network consists of a generator and a discriminator. The generator produces the enhanced image, and the discriminator determines whether the image generated by the generator can be used as the final enhanced image.

[0059] In this invention, the image enhancement model includes a generation layer and a discrimination layer. The generation layer generates a new nighttime video from the original nighttime video, but this new nighttime video undergoes subsequent discrimination to determine whether it can be used as an enhanced nighttime video. The discrimination layer determines whether the new nighttime video meets a similarity condition based on the reflectance brightness of the new nighttime video and the reflectance brightness of the standard white-light image. This similarity condition is whether the similarity between the reflectance brightness of the new nighttime video and the reflectance brightness of the standard white-light image meets a certain range. If it does, the new nighttime video is output as the image-enhanced nighttime video. Each frame in the white-light image and the nighttime video is an image captured from the same scene.

[0060] In this invention, the image enhancement model uses the following formula to determine the enhanced nighttime video:

[0061] ;

[0062] ;

[0063] in, The pixel value of the target point in the reflected image. For each pixel of the target point in the incident image, This represents the pixel value of the target point in the original image.

[0064] The nighttime video monitoring method further provided by this invention can improve the accuracy of identifying various information in the image by performing image enhancement on the original nighttime video, which facilitates the prediction of subsequent abnormal behavior.

[0065] Further inventions of the aforementioned nighttime video monitoring method mainly explain the processing steps for determining semantic features from image-enhanced nighttime videos and determining motion features from original nighttime videos, as detailed below:

[0066] The enhanced nighttime video is input into the SlowFast network model to obtain the semantic and motion features output by the SlowFast network model.

[0067] The SlowFast network model includes a low-frequency channel and a high-frequency channel. The low-frequency channel is used to determine semantic features based on the image-enhanced night video, while the high-frequency channel is used to determine motion features based on the original night video.

[0068] It should be noted that in this invention, climbing is considered a key action for judging abnormal climbing in community theft prevention. Climbing is also a continuous process, and using images alone as the data source for action recognition will lead to significant misjudgments. Therefore, combining motion features is necessary to improve the accuracy of recognition. For this purpose, a SlowFast network model is used to perform action recognition on the surveillance data. This network model consists of two channels: a low-frequency channel, which extracts video frames at a low frequency to detect semantic information from the enhanced nighttime video; and a high-frequency channel, which extracts video frames at a high frequency to detect motion information from the original nighttime video. Separating these channels reduces the computational load of the entire processing.

[0069] In a further invention of the aforementioned nighttime video monitoring method, the main focus is on determining the prediction result based on semantic features and the motion features, and issuing an early warning based on the prediction result, including:

[0070] If continuous abnormal movements are determined based on semantic and motion features, the image-enhanced nighttime video is saved and an early warning message is issued.

[0071] It should be noted that the system analyzes semantic and motion features to determine whether there are continuous abnormal movements in the video, which are considered as illegal intrusion into the monitored area. In this case, the corresponding nighttime video will be saved as evidence, and a warning will be issued to security personnel so that they can take preventive measures in advance.

[0072] In addition, if facial information of individuals making continuous abnormal movements in nighttime videos is obtained based on semantic and motion features, and this facial information is stored in a preset blacklist, an early warning will be issued if the individual re-enters the monitored area, allowing for preventative measures to be taken in advance.

[0073] In a further invention of the aforementioned nighttime video monitoring method, the process for determining the location of early warning points is explained in detail below:

[0074] Determine the image coordinates of the warning point on the nighttime video;

[0075] The image coordinates are converted into the first-world coordinates corresponding to the warning point. The first-world coordinates are the coordinates established with the warning camera as the origin. The night video collected by the warning camera contains the warning point.

[0076] The first-world coordinates are converted into the second-world coordinates corresponding to the early warning points. The second-world coordinates are coordinates established with the monitoring center as the origin.

[0077] Based on the second world coordinates and the world coordinates of multiple preset landmarks, determine multiple relative distances, and then determine the landmark closest to the warning point based on these multiple relative distances.

[0078] It should be noted that in this invention, the warning point is a specific location determined based on the point in the video where an unauthorized intrusion into the monitored area occurs. However, since the monitored area is relatively large, it is necessary to provide security personnel with a range of locations so that they can promptly protect the area surrounding the point where the incident occurred.

[0079] Therefore, the first step is to determine the image coordinates of the warning point on the nighttime video feed. These image coordinates are determined by a certain origin on the video interface, and therefore are not the geographic coordinates of the entire monitored area. Thus, it is necessary to convert the image coordinates into the corresponding first-world coordinates of the warning point. However, these first-world coordinates are geographic coordinates established with the warning camera as the origin, and the nighttime video captured by this camera contains the warning point. Since a monitoring center is set up in the monitored area, which can pinpoint the location of various buildings throughout the area, it is also necessary to convert the first-world coordinates to geographic coordinates with the monitoring center as the origin.

[0080] Therefore, it is necessary to convert the first-world coordinates into the second-world coordinates corresponding to the early warning points. The second-world coordinates are coordinates established with the monitoring center as the origin.

[0081] Then, based on the second world coordinates and the world coordinates of multiple preset landmark buildings, several relative distances are determined. Based on these relative distances, the landmark building closest to the warning point is identified. In essence, this determines the distance between the warning point and recognizable buildings in the monitored area, thereby identifying the standard building closest to the warning point. The information of this building is then output so that security personnel can become familiar with it and establish protective measures in the area between the building and the warning point as soon as possible.

[0082] It should be noted that the conversion from image coordinates to the corresponding first-world coordinates of the warning point includes:

[0083] The first world coordinates corresponding to the warning point are determined using the first formula based on the image coordinates. The first formula is as follows:

[0084] ;

[0085] in, It is First World coordinates. It is a transformation matrix. It's the focal length. It is the length of an image pixel. It is the width of an image in pixels. These are the coordinates of the early warning camera in the image. These are the coordinates of the warning point in the image. It is the depth value of the warning point in the image. It is a translation matrix.

[0086] And converting the first-world coordinates into the second-world coordinates corresponding to the warning point, including:

[0087] The second world coordinates corresponding to the warning point are determined using the second formula based on the first world coordinates. The second formula is as follows:

[0088] ;

[0089] in, These are the coordinates of the monitoring center. It is a second-world coordinate system. It is a first-world coordinate system.

[0090] In a further method of the present invention, by establishing a coordinate system with the monitoring center as the origin, and combining the layout of cameras and surrounding buildings, the warning point can be quickly located in a way that is familiar to the staff.

[0091] The nighttime video monitoring device provided by the present invention is described below. The nighttime video monitoring device described below can be referred to in correspondence with the nighttime video monitoring method described above.

[0092] Figure 2 A schematic diagram of the structure of a nighttime video monitoring device provided by the present invention is shown below. Figure 2 The device includes an enhancement module 21, a determination module 22, and an early warning module 23, wherein:

[0093] Enhancement module 21 is used to process the original night video to obtain an image-enhanced night video;

[0094] The determination module 22 is used to determine semantic features based on the image-enhanced night video and motion features based on the original night video;

[0095] The early warning module 23 is used to determine the prediction result based on semantic features and motion features, and to issue an early warning based on the prediction result. The prediction result indicates whether there is illegal behavior in the night video.

[0096] In a further invention of the aforementioned nighttime video monitoring device, the enhancement module is specifically used for:

[0097] The original nighttime video is input into the image enhancement model to obtain the enhanced nighttime video output by the image enhancement model.

[0098] The image enhancement model includes a generation layer and a discrimination layer. The generation layer is used to generate a new night video from the original night video. The discrimination layer is used to determine whether the new night video meets the similarity condition based on the reflectance brightness of the new night video and the reflectance brightness of the standard white-room image, and then outputs the new night video as the image-enhanced night video. The white-room image and any frame of the night video are images captured from the same scene.

[0099] In a further invention of the aforementioned nighttime video monitoring device, the determining module is specifically used for:

[0100] The enhanced nighttime video is input into the SlowFast network model to obtain the semantic and motion features output by the SlowFast network model.

[0101] The SlowFast network model includes a low-frequency channel and a high-frequency channel. The low-frequency channel is used to determine semantic features based on the image-enhanced night video, and the high-frequency channel is used to determine motion features based on the original night video.

[0102] In a further invention of the aforementioned nighttime video monitoring device, the early warning module is specifically used for:

[0103] If continuous abnormal actions are determined based on the semantic features and motion features, then the image-enhanced nighttime video is saved and a warning message is issued.

[0104] In a further invention of the aforementioned nighttime video monitoring device, the early warning module is also used to: obtain facial information of a person making continuous abnormal movements in the nighttime video based on the semantic features and the motion features, and store the facial information in a preset blacklist.

[0105] In a further invention of the aforementioned nighttime video monitoring device, the device further includes a positioning module for:

[0106] Determine the image coordinates of the warning point on the nighttime video;

[0107] The image coordinates are converted into the first world coordinates corresponding to the warning point. The first world coordinates are coordinates established with the warning camera as the origin. The night video captured by the warning camera contains the warning point.

[0108] The first world coordinates are converted into the second world coordinates corresponding to the warning point, and the second world coordinates are coordinates established with the monitoring center as the origin;

[0109] Based on the second world coordinates and the world coordinates of multiple preset landmarks, determine multiple relative distances, and then determine the landmark closest to the warning point based on these multiple relative distances.

[0110] In a further invention of the aforementioned nighttime video monitoring device, the positioning module, during the process of converting the image coordinates into the first world coordinates corresponding to the warning point, is specifically used for:

[0111] The first world coordinates corresponding to the warning point are determined using a first formula based on the image coordinates. The first formula is as follows:

[0112] ;

[0113] in, It is First World coordinates. It is a transformation matrix. It's the focal length. It is the length of an image pixel. It is the width of an image in pixels. These are the coordinates of the early warning camera in the image. These are the coordinates of the warning point in the image. It is the depth value of the warning point in the image. It is a translation matrix.

[0114] In a further invention of the aforementioned nighttime video monitoring device, the positioning module, during the process of converting the first world coordinates into the second world coordinates corresponding to the warning point, is specifically used for:

[0115] The second world coordinates corresponding to the warning point are determined using a second formula based on the first world coordinates. The second formula is as follows:

[0116] ;

[0117] in, These are the coordinates of the monitoring center. It is a second-world coordinate system. It is a first-world coordinate system.

[0118] Since the device described in this embodiment of the invention is based on the same principle as the method described in the above embodiments, more detailed explanations will not be repeated here.

[0119] It should be noted that, in the embodiments of the present invention, the relevant functional modules can be implemented by a hardware processor.

[0120] The nighttime video monitoring device provided by this invention extracts semantic features from the enhanced nighttime video and motion features from the original nighttime video, and combines the semantic features and motion features to accurately identify the target that triggers the warning, avoiding false alarms and reducing the need for additional auxiliary equipment, thus saving construction costs.

[0121] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3As shown, the electronic device may include a processor 31, a communication interface 32, a memory 33, and a communication bus 34, wherein the processor 31, the communication interface 32, and the memory 33 communicate with each other via the communication bus 34. The processor 31 can call a computer program in the memory 33 to execute the steps of a nighttime video monitoring method, such as: processing the original nighttime video to obtain an image-enhanced nighttime video; determining semantic features based on the image-enhanced nighttime video and determining motion features based on the original nighttime video; determining a prediction result based on the semantic features and motion features; and issuing an early warning based on the prediction result, wherein the prediction result indicates whether there is illegal behavior in the nighttime video.

[0122] Furthermore, the logical instructions in the aforementioned memory 33 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0123] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, which, when executed by a computer, enable the computer to perform the steps of the nighttime video monitoring methods provided by the above methods, for example including: processing the original nighttime video to obtain an image-enhanced nighttime video; determining semantic features based on the image-enhanced nighttime video, and determining motion features based on the original nighttime video; determining a prediction result based on the semantic features and motion features, and issuing an early warning based on the prediction result, wherein the prediction result characterizes whether there is illegal behavior in the nighttime video.

[0124] On the other hand, embodiments of this application also provide a processor-readable storage medium storing a computer program for causing the processor to execute the steps of the methods provided in the above embodiments, such as: processing the original night video to obtain an image-enhanced night video; determining semantic features based on the image-enhanced night video and determining motion features based on the original night video; determining a prediction result based on the semantic features and motion features; and issuing an early warning based on the prediction result, wherein the prediction result characterizes whether there is illegal behavior in the night video.

[0125] The processor-readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic memory (e.g., floppy disk, hard disk, magnetic tape, magneto-optical disk (MO)), optical memory (e.g., CD, DVD, BD, HVD), and semiconductor memory (e.g., ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)).

[0126] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0127] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A nighttime video monitoring method, characterized in that, include: The original nighttime video is processed to obtain an image-enhanced nighttime video; Semantic features are determined from the image-enhanced nighttime video, and motion features are determined from the original nighttime video; The prediction result is determined based on the semantic features and the motion features, and an early warning is issued based on the prediction result, wherein the prediction result indicates whether there is illegal behavior in the night video; The method further includes the step of determining the location of the warning point, including: Determine the image coordinates of the warning point on the nighttime video; the warning point is a specific location determined based on the point in the video where an illegal intrusion into the monitored area occurs. The image coordinates are converted into the first world coordinates corresponding to the warning point. The first world coordinates are coordinates established with the warning camera as the origin. The night video captured by the warning camera contains the warning point. The first world coordinates are converted into the second world coordinates corresponding to the warning point, and the second world coordinates are coordinates established with the monitoring center as the origin; Based on the second world coordinates and the world coordinates of multiple preset landmarks, determine multiple relative distances, and then determine the landmark closest to the warning point based on these multiple relative distances.

2. The nighttime video monitoring method according to claim 1, characterized in that, The process of processing the original nighttime video to obtain an image-enhanced nighttime video includes: The original nighttime video is input into the image enhancement model to obtain the enhanced nighttime video output by the image enhancement model. The image enhancement model includes a generation layer and a discrimination layer. The generation layer is used to generate a new night video from the original night video. The discrimination layer is used to determine whether the new night video meets the similarity condition based on the reflectance brightness of the new night video and the reflectance brightness of the standard white-room image, and then outputs the new night video as the image-enhanced night video. The white-room image and any frame of the night video are images captured from the same scene.

3. The nighttime video monitoring method according to claim 1, characterized in that, The process of determining semantic features based on the image-enhanced nighttime video and determining motion features based on the original nighttime video includes: The enhanced nighttime video is input into the SlowFast network model to obtain the semantic and motion features output by the SlowFast network model. The SlowFast network model includes a low-frequency channel and a high-frequency channel. The low-frequency channel is used to determine semantic features based on the image-enhanced night video, and the high-frequency channel is used to determine motion features based on the original night video.

4. The nighttime video monitoring method according to claim 1, characterized in that, The step of determining the prediction result based on the semantic features and the motion features, and issuing an early warning based on the prediction result, includes: If continuous abnormal actions are determined based on the semantic features and motion features, then the image-enhanced nighttime video is saved and a warning message is issued.

5. The nighttime video monitoring method according to claim 1, characterized in that, The method further includes: obtaining facial information of individuals making continuous abnormal movements in nighttime videos based on the semantic features and the motion features, and storing the facial information in a preset blacklist.

6. The nighttime video monitoring method according to claim 1, characterized in that, The step of converting the image coordinates into the first world coordinates corresponding to the warning point includes: The first world coordinates corresponding to the warning point are determined using a first formula based on the image coordinates. The first formula is as follows: ； in, It is First World coordinates. It is a transformation matrix. It's the focal length. It is the length of an image pixel. It is the width of an image in pixels. These are the coordinates of the early warning camera in the image. These are the coordinates of the warning point in the image. It is the depth value of the warning point in the image. It is a translation matrix.

7. The nighttime video monitoring method according to claim 1, characterized in that, The step of converting the first world coordinates into the second world coordinates corresponding to the warning point includes: The second world coordinates corresponding to the warning point are determined using a second formula based on the first world coordinates. The second formula is as follows: ； in, These are the coordinates of the monitoring center. It is a second-world coordinate system. It is a first-world coordinate system.

8. A nighttime video monitoring device, characterized in that, include: The enhancement module is used to process the original night video to obtain an enhanced night video. The determination module is used to determine semantic features based on the image-enhanced night video and motion features based on the original night video; The early warning module is used to determine the prediction result based on the semantic features and the motion features, and to issue an early warning based on the prediction result, wherein the prediction result indicates whether there is illegal behavior in the night video; The device further includes a positioning module for: Determine the image coordinates of the warning point on the nighttime video; the warning point is a specific location determined based on the point in the video where an illegal intrusion into the monitored area occurs. The image coordinates are converted into the first world coordinates corresponding to the warning point. The first world coordinates are coordinates established with the warning camera as the origin. The night video captured by the warning camera contains the warning point. The first world coordinates are converted into the second world coordinates corresponding to the warning point, and the second world coordinates are coordinates established with the monitoring center as the origin; Based on the second world coordinates and the world coordinates of multiple preset landmarks, determine multiple relative distances, and then determine the landmark closest to the warning point based on these multiple relative distances.

9. An electronic device comprising a processor and a memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the nighttime video monitoring method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the nighttime video monitoring method as described in any one of claims 1 to 7.

Images