Warning method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a warning method and device, electronic equipment and a storage medium. According to the embodiment of the invention, an observation video can be obtained, the observation video comprises a picture currently observed in a target area, and the target area comprises a forbidden area and an open area; object recognition is carried out on the observation video to obtain a target object, and the target object is located in the forbidden zone in the observation video; and performing warning processing on the target object in the target area. The embodiment of the invention can be applied to the traffic field. According to the embodiment of the invention, the real-time observation video of the target area is automatically collected, and the image of the real-time observation video is subjected to object recognition, so that the target objects in the forbidden area are recognized, and the target objects are warned. Compared with manual patrol and warning, the scheme provided by the embodiment of the invention can reduce the labor cost, thereby improving the warning efficiency.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a warning method, device, electronic device and storage medium. Background Art

[0002] Self-driving in scenic spots is a very flexible and personalized way of traveling. Tourists can freely plan their itineraries according to their own interests and schedules. However, tourists may behave uncivilized while driving in scenic spots, such as crushing lawns and vegetation, driving into prohibited areas such as pedestrian areas, nature reserves, and pastoral areas, which poses a serious threat to the safety of other tourists and the natural environment.

[0003] At present, in order to promptly discover and prevent such behavior, scenic spots often set up patrol personnel and formulate relevant regulations and punishment measures. For large scenic spots, a large number of patrol personnel need to be set up and each person's patrol route, shift time, etc. are specified. Therefore, for large scenic spots, a large amount of manpower costs need to be invested and patrols are often not timely and comprehensive. Therefore, the current warning method is inefficient. Summary of the invention

[0004] The embodiments of the present application provide a warning method, device, electronic device and storage medium, which can improve the efficiency of the warning method.

[0005] The present application embodiment provides a warning method, including:

[0006] Obtaining observation video, the observation video includes the currently observed images of the target area, and the target area includes the restricted area and the open area;

[0007] Performing object recognition on the observation video to obtain the target object, where the target object is located in the restricted area in the observation video;

[0008] Provide warning to target objects in the target area.

[0009] The present application also provides a warning device, including:

[0010] A video unit is used to obtain an observation video, where the observation video includes a currently observed image of a target area, where the target area includes a restricted area and an open area;

[0011] The recognition unit is used to perform object recognition on the observation video to obtain a target object, where the target object is located in the restricted area in the observation video;

[0012] The warning unit is used to perform warning processing on the target object in the target heap area.

[0013] In some embodiments, obtaining the observed video includes:

[0014] Obtain observation videos collected in the target area;

[0015] Display the images of the target area in the observation video;

[0016] In response to the area setting operation for the screen, a restricted area and an open area of ​​the target area are determined in the screen.

[0017] In some embodiments, the area setting operation includes a segmentation operation, and in response to the area setting operation on the picture, determining the restricted area and the open area of ​​the target area in the picture includes:

[0018] In response to a split operation on the screen, a split line corresponding to the split operation is displayed in the screen;

[0019] The target area in the image is segmented based on the segmentation line to obtain a restricted area and an open area.

[0020] In some embodiments, the area setting operation includes a restricted area drawing operation. In response to the area setting operation on the screen, determining a restricted area and an open area of ​​the target area in the screen includes:

[0021] In response to a restricted area drawing operation on the screen, displaying the drawn restricted area sub-area in the screen;

[0022] All restricted area sub-areas are combined to obtain the restricted area of ​​the target area;

[0023] Determine the open area, which is the area in the target area except the restricted area.

[0024] In some embodiments, determining a restricted area and an open area of ​​a target area in a picture includes:

[0025] A neural network is used to segment the image and determine the restricted area and open area of ​​the target area in the image.

[0026] In some embodiments, the target object performs an alert process, including:

[0027] A warning message is sent to a warning device, the warning device is installed in a target area, and the warning message carries a message for warning the target object, so that the warning device broadcasts the warning message in the target area.

[0028] In some embodiments, after performing object recognition on the observed video and obtaining the target object, the method further includes:

[0029] Identify the target object and obtain the identity information of the target object;

[0030] Conduct warnings on target objects in the target area, including:

[0031] Based on the identity information of the target object, the target object is warned in the target area.

[0032] In some embodiments, the target object includes a target vehicle, the identity information includes license plate information, and performing identity recognition on the target object to obtain the identity information of the target object includes:

[0033] Locate the license plate position of the target vehicle in the observation video;

[0034] Text recognition is performed based on the license plate position to obtain the license plate information of the target vehicle.

[0035] In some embodiments, after performing object recognition on the observed video and obtaining the target object, the method further includes:

[0036] When displaying the observation video, an outer frame is added to the target object, and the outer frame is used to highlight the target object in the observation video.

[0037] In some embodiments, after performing object recognition on the observed video and obtaining the target object, the method further includes:

[0038] When the target object is located in the restricted area for the first time, the first entry time of the target object is recorded;

[0039] When the target object is not in the restricted area, the departure time of the target object is recorded;

[0040] Based on the first entry time and the exit time, the residence time of the target object is determined, and the residence time represents the time the target object stays in the restricted area.

[0041] In some embodiments, after performing object recognition on the observed video and obtaining the target object, the method further includes:

[0042] Obtain the target object's behavior information during the detention time;

[0043] Determine the target's violation based on behavioral information;

[0044] Conduct warnings on target objects in the target area, including:

[0045] Based on the target object's illegal behavior, the target object is warned in the target area.

[0046] In some embodiments, after determining the illegal behavior of the target object based on the behavior information, the method further includes:

[0047] The residence time, behavior information, and violation behavior of the target object are sent to the mobile terminal so that the mobile terminal can view the target object, as well as the residence time, behavior information, and violation behavior of the target object.

[0048] An embodiment of the present application also provides an electronic device, including a memory storing multiple instructions; the processor loads instructions from the memory to execute the steps of any warning method provided in the embodiment of the present application.

[0049] An embodiment of the present application also provides a computer-readable storage medium, which stores a plurality of instructions, and the instructions are suitable for a processor to load to execute the steps of any one of the warning methods provided in the embodiment of the present application.

[0050] The embodiment of the present application can obtain an observation video, which includes a currently observed image of a target area, and the target area includes a restricted area and an open area; perform object recognition on the observation video to obtain a target object, and the target object is located in the restricted area in the observation video; and perform warning processing on the target object in the target area.

[0051] The present application automatically collects real-time observation videos of the target area and performs object recognition on the images, thereby identifying target objects located in the restricted area, so as to automatically warn these target objects. Therefore, compared with manual patrols and warnings, the embodiments of the present application can reduce labor costs and improve warning efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0053] Figure 1a It is a scene schematic diagram of the warning method provided in the embodiment of the present application;

[0054] Figure 1b It is a flowchart of the warning method provided in the embodiment of the present application;

[0055] Figure 1c This is a schematic diagram of a region setting page of a warning method provided in an embodiment of the present application;

[0056] Figure 1d It is a segmentation schematic diagram of the warning method provided in the embodiment of the present application;

[0057] Figure 1e It is a schematic diagram of the warning method provided in the embodiment of the present application;

[0058] Figure 1f This is a schematic diagram of object recognition of the warning method provided in the embodiment of the present application;

[0059] Figure 2a It is a flowchart of the warning method provided in the embodiment of the present application applied in a scenic spot scene;

[0060] Figure 2b Schematic diagram of a video acquisition module of the warning method provided in an embodiment of the present application;

[0061] Figure 3 is a schematic diagram of the structure of the warning device provided in the embodiment of the present application;

[0062] Figure 4 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0064] Embodiments of the present application provide a warning method, device, electronic device, and storage medium.

[0065] The warning device can be integrated into an electronic device, which can be a terminal, a server, etc. The terminal can be a mobile phone, a tablet computer, a smart Bluetooth device, a laptop, or a personal computer (PC); the server can be a single server or a server cluster composed of multiple servers.

[0066] In some embodiments, the warning device may also be integrated into multiple electronic devices. For example, the warning device may be integrated into multiple servers, and the warning method of the present application may be implemented by multiple servers.

[0067] In some embodiments, the server may also be implemented in the form of a terminal.

[0068] For example, refer to Figure 1a The target area includes a restricted area and an open area. The camera located in the target area can collect observation videos. The electronic device can obtain the observation videos from the camera, and the observation videos include the pictures currently observed in the target area; perform object recognition on the observation video to obtain the target object, and the target object is located in the restricted area in the observation video; use the sound located in the target area to warn the target object, so as to stop the target object from continuing to stay in the restricted area.

[0069] It should be noted that the serial numbers of the following embodiments are not intended to limit the preferred order of the embodiments.

[0070] Artificial Intelligence (AI) is a technology that uses digital computers to simulate human perception of the environment, acquire knowledge, and use knowledge. This technology can enable machines to have functions similar to human perception, reasoning, and decision-making. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation, and other major directions.

[0071] Intelligent Traffic System (ITS), also known as Intelligent Transportation System, is an effective and comprehensive application of advanced science and technology (information technology, computer technology, data communication technology, sensor technology, electronic control technology, automatic control theory, operations research, artificial intelligence, etc.) to transportation, service control and vehicle manufacturing, strengthening the connection between vehicles, roads and users, thus forming a comprehensive transportation system that ensures safety, improves efficiency, improves the environment and saves energy. Or Intelligent Vehicle Infrastructure Cooperative System (IVICS), referred to as vehicle-infrastructure cooperative system, is a development direction of intelligent transportation system (ITS). Vehicle-infrastructure cooperative system adopts advanced wireless communication and new generation Internet technologies to implement dynamic real-time information interaction between vehicles and roads in all directions, and carries out active vehicle safety control and road cooperative management based on the collection and integration of dynamic traffic information in all time and space, fully realizing the effective coordination of people, vehicles and roads, ensuring traffic safety and improving traffic efficiency, thus forming a safe, efficient and environmentally friendly road traffic system.

[0072] Among them, computer vision (CV) can be applied to intelligent transportation systems. Computer vision is a technology that uses computers to replace human eyes to identify, measure and further process target images. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, virtual reality, augmented reality, simultaneous positioning and map construction, automatic driving, intelligent transportation and other technologies, and also includes common biometric recognition technologies such as face recognition and fingerprint recognition. For example, image processing technologies such as image coloring and image stroke extraction.

[0073] Speech technology can be applied to intelligent transportation systems. The key technologies of speech technology include automatic speech recognition technology, speech synthesis technology and voiceprint recognition technology. Enabling computers to listen, see, speak and feel is the future development direction of human-computer interaction, among which speech has become one of the most promising human-computer interaction methods in the future.

[0074] In this embodiment, a warning method based on a smart traffic system involving artificial intelligence is provided, such as Figure 1b As shown, the specific process of the warning method can be as follows:

[0075] 101. Obtain an observation video, where the observation video includes a currently observed image of a target area, and the target area includes a restricted area and an open area.

[0076] The target area may be one of many areas, and an area refers to a relatively large area with certain uniformity or specific attributes in some aspects. For example, an area may refer to a scenic spot, a geographical area, an administrative area, a community, a transportation network area, etc.

[0077] Among them, restricted areas refer to areas where vehicles and pedestrians are prohibited or restricted from entering due to certain special reasons, such as safety, environmental protection, cultural heritage protection, etc.; open areas are areas where vehicles and pedestrians are allowed to pass freely, such as roads, leisure areas and commercial areas.

[0078] The observed video refers to the video of part or all of the target area currently observed, so the picture of the target area may include part or all of the current pictures of the restricted area and the open area.

[0079] In some embodiments, after the observation video is acquired, the observation video may also be played and displayed so that the user can observe the current situation of the target area.

[0080] In some embodiments, the observed video that has been played can also be played back.

[0081] In some embodiments, in response to the user's local zoom-in operation, a local zoom-in point in the observation video screen can be determined, and the observation video can be zoomed in with the local zoom-in point as the center, so that the user can manually zoom in the observation video to observe the detailed picture when observing abnormal conditions in the target area.

[0082] In some embodiments, the camera's acquisition parameters such as acquisition angle, direction, focal length, exposure, etc. can be controlled in response to the user's remote control operation on the camera, so that the user can adjust and change the image captured by the camera.

[0083] There are many ways to obtain observation videos, for example, they can be pulled from a video database, collected from a camera, and so on.

[0084] In some embodiments, the camera can be installed in the target area, and can be installed at a high position in the target area to collect the current image of the target area from a bird's-eye view, or can be installed at a low position in the target area to collect the current image of the target area from a level view. The specific installation height is not limited and can be set according to actual needs.

[0085] In some embodiments, it is necessary to calibrate the restricted area and the open area in the observed video screen so that the server can recognize the restricted area and the open area in the observed video screen. Therefore, step 101 may include the following steps:

[0086] A1. Obtain observation videos collected in the target area;

[0087] A2. Display the images of the target area in the observation video;

[0088] A3. In response to the area setting operation for the screen, a restricted area and an open area of ​​the target area are determined in the screen.

[0089] The calibration, i.e., area setting operation, may include manual calibration or automatic identification calibration. Therefore, in some embodiments, an area setting page may be provided in step A3, the area setting page including a screen, and through the area setting operation on the area setting page, the restricted area and the open area of ​​the target area are determined in the screen of the area setting page.

[0090] refer to Figure 1c In some embodiments, the area setting page may also include an alert setting control for setting, including but not limited to, the alert effective time, reminder method, risk level, and object type of the target object.

[0091] The warning effective time refers to the time when step 103 takes effect, for example, Figure 1c The warning effective time set in is 8:00:00 start time and 18:00:00 end time. Therefore, the target object will be warned in the target area from 8:00:00 to 18:00:00 every day, and no warning will be given at other times.

[0092] The reminder method refers to the reminder method for the user after the target object is found in step 102, for example Figure 1c The reminder method set in is a mobile pop-up window. Therefore, when the target object is found, a reminder pop-up window can be popped up on the mobile terminal carried by users such as scenic area staff, security personnel, etc. to remind users that the target object has been found.

[0093] The risk level refers to the risk level brought by the behavior of the target object. In some embodiments, step 102 can identify the behavior of the target object in the picture to obtain the risk level of its behavior. When the risk level reaches or exceeds the set level, the user is reminded. Figure 1c When the risk level reaches or exceeds the general level, the mobile terminal carried by the user can pop up a reminder window to remind the user to stop the target object from engaging in general or higher level risk behaviors.

[0094] The object type of the target object may include multiple types, for example, Figure 1c ,The object type of the target object can be vehicle, person, etc.

[0095] Therefore, the calibration, i.e., area setting operation, may include manual calibration, for example, manually drawing one or more line segments in the picture to outline the restricted area or open area in the picture. Therefore, in some embodiments, the area setting operation includes a segmentation operation, and step A3 may include the following steps:

[0096] In response to a split operation on the screen, a split line corresponding to the split operation is displayed in the screen;

[0097] The target area in the image is segmented based on the segmentation line to obtain the restricted area and the open area.

[0098] For example, refer to Figure 1d When the user draws a dividing line on the screen of the area setting page, the screen can be divided into a restricted area and an open area based on the dividing line.

[0099] Therefore, in some embodiments, the calibration, i.e., area setting operation, may include manual calibration, the area setting operation includes a restricted area drawing operation, and step A3 may include the following steps:

[0100] In response to a restricted area drawing operation on the screen, displaying the drawn restricted area sub-area in the screen;

[0101] All restricted area sub-areas are combined to obtain the restricted area of ​​the target area;

[0102] Determine the open area, which is the area in the target area except the restricted area.

[0103] For example, refer to Figure 1e , after the user draws the restricted area sub-area on the screen of the area setting page, these restricted area sub-areas can be taken as the union to finally obtain the restricted area.

[0104] Similarly, in some embodiments, the area setting operation includes an open area drawing operation, and step A3 may include the following steps:

[0105] In response to an open area drawing operation on the screen, displaying the drawn open sub-area in the screen;

[0106] All open sub-areas are combined to obtain the open area of ​​the target area;

[0107] Determine the restricted area, which is the area in the target area except the open area.

[0108] In some embodiments, the automatic identification and calibration can be implemented by using an artificial neural network, for example, identifying the restricted area and the open area in the picture by using an artificial neural network. Therefore, in some embodiments, step A3 can include the following steps:

[0109] A neural network is used to segment the image and determine the restricted area and open area of ​​the target area in the image.

[0110] Among them, region segmentation is an important task in the field of computer vision, which aims to divide the regions in an image or video into parts with specific semantics. There are many region segmentation methods, such as semantic segmentation, which divides the image into regions with the same semantics, and each pixel is assigned to a restricted area or an open area; instance segmentation is similar to semantic segmentation, but it not only marks the category to which the pixel belongs, but also distinguishes different instances, such as vehicle instances, person instances, non-motor vehicle instances, building instances, etc.

[0111] 102. Perform object recognition on the observation video to obtain a target object, where the target object is located in a restricted area in the observation video.

[0112] In some embodiments, a neural network may be used to perform object recognition on the observed video to obtain multiple candidate objects, and the candidate objects located in the restricted area of ​​the picture are determined as target objects.

[0113] Among them, the candidate objects can be vehicles, non-motor vehicles, people, etc.

[0114] For example, in some embodiments, Faster RCNN (Region-based Convolutional Neural Network) can be used for object recognition. Faster RCNN is a deep learning model for target detection, including convolutional layers (Conv layers), Region Proposal Networks (RPN), Region of Interest Pooling (Roi Pooling) and Classification.

[0115] For example, refer to Figure 1f For the observed video screen, first scale it to a fixed size to get a scaled image, and then send the scaled image to the convolution layer to extract the feature map; RoI pooling is used to combine the feature map and the proposed region; RPN is used to generate an anchor box in the feature map, which is cropped and filtered to determine whether its content is an object, and the bounding box regression is used to correct the anchor box to form a more accurate proposed region; then, RoI pooling extracts the proposed region feature map from the feature maps and the proposed region, and sends the proposed region feature map to the fully connected layer (FC Layer) to determine the category to which the proposed region feature map belongs; finally, the object classifier is used for probabilistic classification and the bounding box regression is used for joint training to improve the accuracy.

[0116] In some embodiments, after obtaining the target object, a local zoom operation can be automatically performed based on the target object in the observed video screen, that is, the target object in the screen is used as the local zoom point, and the observed video is zoomed in with the local zoom point as the center, so that the user can further observe the enlarged target object in the screen.

[0117] In some embodiments, after obtaining the target object, the camera can be controlled to automatically aim at the target object in the target area, and the camera can be controlled to adjust its focal length so that the image of the target object in the captured image is focused and magnified, thereby realizing a focusing and magnifying operation, so that the user can further observe the enlarged target object in the image.

[0118] In some embodiments, the target object may be accurately warned based on the identity information of the target object, and the following steps may be included after step 102:

[0119] Identify the target object and obtain the identity information of the target object;

[0120] Step 103 may include the following steps:

[0121] Based on the identity information of the target object, the target object is warned in the target area.

[0122] For example, the license plate of the target vehicle is identified, and when the warning voice is broadcast, the license plate number of the target vehicle is broadcast in the warning voice, thereby achieving accurate warning for vehicles that illegally enter the restricted area.

[0123] In some embodiments, the target object includes a target vehicle, the identity information includes license plate information, and identifying the target object to obtain the identity information of the target object may include the following steps:

[0124] Locate the license plate position of the target vehicle in the observation video;

[0125] Text recognition is performed based on the license plate position to obtain the license plate information of the target vehicle.

[0126] For example, based on the proposed area of ​​the target vehicle collected by the above-mentioned Faster RCNN, the license plate position is located therefrom; the image at the license plate position is segmented according to Chinese characters, letters and numbers to obtain segmented images, and then the segmented images are respectively input into the license plate recognition model for recognition, and finally the output results are sequentially combined into the license plate number output.

[0127] Among them, the license plate recognition model can be an artificial neural network. Since the images collected by the camera during actual use are affected by objective factors such as light and angle, the observed video may have problems such as unclear license plate images and partial license plate obstruction, thereby causing false detection. Therefore, in some embodiments, in the training stage of the license plate recognition model, in addition to training the conventional number and license plate letter recognition, the grassland environment, the grassland background color, vehicle angle, light intensity, rain cover and other conditions can also be simulated, that is, a variety of observation videos with different grassland environments, grassland background color, vehicle angle, light intensity, rain cover and other conditions are collected as training samples to train the license plate recognition model, thereby improving the recognition accuracy of the algorithm.

[0128] In some embodiments, the following steps may also be included after step 102:

[0129] When displaying the observation video, an outer frame is added to the target object, and the outer frame is used to highlight the target object in the observation video.

[0130] The outer frame may be the proposed area of ​​the target vehicle collected by the Faster RCNN, or may be an anchor frame obtained in the process.

[0131] In some embodiments, the time and behavior of the target object in the restricted area can also be recorded, so as to evaluate the severity of the target object when the target object is warned in step 103, and to facilitate the warning and punishment based on the evidence. Therefore, the following steps can be included after step 102:

[0132] When the target object is located in the restricted area for the first time, the first entry time of the target object is recorded;

[0133] When the target object is not in the restricted area, the departure time of the target object is recorded;

[0134] Based on the first entry time and the exit time, the residence time of the target object is determined, and the residence time represents the time the target object stays in the restricted area.

[0135] In some embodiments, the following steps may also be included after step 102:

[0136] Obtain the target object's behavior information during the detention time;

[0137] Determine the target's violation based on behavioral information;

[0138] Conduct warnings on target objects in the target area, including:

[0139] Based on the target object's illegal behavior, the target object is warned in the target area.

[0140] The behavior information may include movement speed, movement direction, posture, action, etc. In some embodiments, the behavior type may include speeding behavior, littering behavior, illegal fire-making behavior, etc.

[0141] In some embodiments, a behavior recognition model may be used to determine the target object's illegal behavior based on the behavior information, such as Temporal Convolutional Networks (TCN), Recurrent Neural Networks (RNN), or Long Short-Term Memory (LSTM), which can process sequence data and are suitable for time correlation in videos.

[0142] In some embodiments, after determining the illegal behavior of the target object based on the behavior information, the method further includes:

[0143] The residence time, behavior information, and violation behavior of the target object are sent to the mobile terminal so that the mobile terminal can view the target object, as well as the residence time, behavior information, and violation behavior of the target object.

[0144] For example, the time when the target vehicle enters and leaves the restricted area is recorded, and a violation event is generated based on its license plate number. The detention time, behavior information, and violation of the target object are sent to the staff's mobile terminal. The staff can evaluate the severity of the target vehicle based on the data of the event and impose reasonable penalties on the violation if necessary.

[0145] For example, behavior information may include vehicle speed and travel distance, and violations may include vegetation damage area calculated based on vehicle speed and travel distance. Staff can punish relevant violators based on travel speed, travel distance, length and area of ​​grassland damage, and combined with comprehensive factors such as the location in the grassland, the length of time it appears, and the degree of grassland damage.

[0146] In some embodiments, illegal behaviors may also include illegal fire making, creation of abnormal smoke and other abnormal events. When these abnormal events are identified in step 102, warnings may also be issued for these abnormal events in step 103 to protect the safety of personnel and the environment.

[0147] 103. Provide warnings to target objects in the target area.

[0148] In some embodiments, the target object can be warned in the target area by installing warning equipment such as broadcasting, electronic bulletin boards, large screens, etc. Step 103 may include the following steps:

[0149] A warning message is sent to the warning device, the warning message carrying a message for warning the target object, so that the warning device broadcasts the warning message in the target area.

[0150] For example, a warning message may be generated based on the image, identity information, residence time, behavior information and / or violation of the target object obtained in step 102, and the warning message may be broadcasted in the target area using a warning device.

[0151] For example, the warning message may be a voice message, and the voice message may have a fixed template. The warning message may be obtained by filling the license plate number of the target vehicle into the fixed template, and the broadcast may play the warning message to the target area.

[0152] For example, the warning message may be an image message, which may display a picture of the target vehicle in the observation video and a preset warning text, such as "Please leave the restricted area for the vehicle in the video". In some embodiments, the preset warning text may also include the license plate number of the target vehicle, such as "Please leave the restricted area for the vehicle A123456 in the video".

[0153] From the above, it can be seen that the embodiment of the present application can obtain an observation video, which includes the currently observed image of the target area, and the target area includes a restricted area and an open area; perform object recognition on the observation video to obtain the target object, and the target object is located in the restricted area in the observation video; and perform warning processing on the target object in the target area.

[0154] Therefore, the solution can automatically collect real-time observation videos of the target area and perform object recognition on the images, thereby identifying the target objects in the restricted area so as to warn these target objects. Compared with manual patrols and warnings, the embodiments of the present application can reduce labor costs and thus improve warning efficiency.

[0155] The method described in the above embodiment will be further described in detail below.

[0156] In this embodiment, a scenic spot scene is taken as an example to describe the method of the embodiment of the present application in detail.

[0157] like Figure 2a As shown, the specific process of a warning method is as follows:

[0158] (a) Video acquisition module.

[0159] refer to Figure 2b The video acquisition module integrates various applications, data and systems through IPaaS (Integration Platform as a Service) to ensure that they can work together and integrate seamlessly; the IOT (Internet of Things) gateway is used as the video gateway to collect data from sensors, cameras and other devices in the scenic area; the image data collected from cameras and other devices is processed and converted through image data conversion, such as compression, feature extraction or format conversion, so as to perform more advanced analysis and recognition.

[0160] IPaaS is a cloud service model that provides integrated tools and services for connecting different applications, data, and systems. It allows organizations to easily integrate various cloud services, local systems, and third-party applications to achieve seamless connection of data flows and business processes. The goal of IPaaS is to simplify the integration process, reduce development and deployment time, and increase business flexibility.

[0161] An IoT gateway is a key component in an Internet of Things (IoT) system that connects and coordinates communications between IoT devices and cloud platforms.

[0162] The video acquisition module supports real-time video display on the client side and can playback the video. The video acquisition module can pull observation videos of multiple code streams and broadcast them to implement warning processing.

[0163] The video acquisition module uses the video and frame data collected by the camera as the analysis image, and the subsequent algorithm analysis also uses the video image of the video acquisition module as the basis for image recognition analysis and behavior recognition. The video acquisition module supports manual selection of focus points and one-click zooming in when viewing, and also supports automatic linkage of the camera to adjust the focus and viewing direction to zoom in on abnormal areas when violations or abnormal events are detected.

[0164] The video gateway can obtain the device status of each camera through the device status process, such as device parameters such as focal length, direction, and posture. The observation video collected by each camera can be obtained through the video acquisition process; the message process can generate warning messages.

[0165] The video gateway's gateway software development kit (SDK) provides a range of functions for:

[0166] Message detection: The SDK provides the ability to detect data or messages sent from devices, sensors, or other components to the gateway. Message detection allows applications to be notified or trigger corresponding processing when a message arrives.

[0167] Registration and Login: The SDK provides registration and login functions to ensure that the application can establish effective communication with the gateway.

[0168] Message callback: The SDK supports message callback, that is, when a message of a specific type arrives, the SDK can call the registered callback function.

[0169] Message sending: The SDK provides the message sending function, which enables applications to send messages to other devices, services, or systems.

[0170] The master-slave relationship of the video gateway is a relationship between a master gateway (Master Gateway) and multiple slave gateways (Slave Gateway). The following is an explanation of the four steps involved in these master-slave relationships:

[0171] In the establishment of the master-slave relationship, the master gateway must first actively detect available slave gateways on the network.

[0172] Multicast Monitoring: Once a slave gateway is detected, the master gateway can use the multicast monitoring mechanism to detect the slave gateway availability, performance, and other key indicators in real time to more effectively monitor and manage multiple slave gateways.

[0173] Role Assignment: In a master-slave relationship, each gateway may be assigned different tasks. The master gateway is responsible for overall control and coordination, while the slave gateway is responsible for performing some specific tasks or providing additional services.

[0174] Voting: The voting mechanism can verify the status of each gateway through communication between the master and slave gateways, and make decisions based on the majority of voting results, thereby ensuring reliability and fault tolerance in the master-slave relationship.

[0175] (ii) Object recognition module.

[0176] The object recognition module can set restricted areas in scenic spots on the client and identify target objects that have invaded restricted areas. The inter-frame difference method can be used to obtain moving targets in restricted areas:

[0177]

[0178] Among them, f k (i,j) is the i-th horizontal pixel and the j-th vertical pixel of the k-th frame image, f k-1 (i, j) is the i-th horizontal and j-th vertical pixel of the k-1 frame image, Th is the threshold, and d(i, j) is a binary image.

[0179] The object recognition module sets the target object on the client, such as setting the target object to a vehicle entering a restricted area, a person entering a restricted area, or a vehicle and a person entering a restricted area.

[0180] There are two ways to check the restricted area:

[0181] 1. Linear selection: Use one or more lines to separate the restricted area of ​​the scenic area.

[0182] 2. Area selection: merge one or more sub-areas into a restricted area.

[0183] (3) Illegal object identification module.

[0184] The illegal object recognition module can perform target recognition based on the Faster RCNN neural network convolution model. If the illegal target object is identified as a vehicle, the license plate of the illegal vehicle can be recognized.

[0185] (IV) Violation details identification module.

[0186] The violation details recognition module may include multiple functions such as vehicle detection, image acquisition, and license plate recognition. The violation details recognition module locates the license plate from the image of the illegal vehicle based on the image of the illegal vehicle; the acquired license plate image is segmented according to Chinese characters, letters, and numbers, and the segmented images are input into the violation details recognition module for recognition, and finally the output results are sequentially combined into the license plate number output.

[0187] After identifying an illegal vehicle, the client can control the staff's pop-up reminder. The staff can control the scenic area loudspeaker to remotely shout according to the license plate number. The shouting with license plate information is more deterrent than ordinary prompts. If necessary, the vehicle information can be obtained based on the license plate linkage background for further expulsion.

[0188] In addition, the illegal vehicles can be framed in the observation video to show the scene more clearly. When the illegal vehicle is detected to enter the restricted area, the intrusion time is recorded and the video recording of the illegal vehicle is turned on; when the illegal vehicle leaves the restricted area, a second record is made to record its departure time; after the illegal vehicle leaves the restricted area, the detention time is automatically calculated and a violation event is generated based on the license plate number. The staff can evaluate the severity of the violation event and impose reasonable and well-founded penalties on the illegal vehicle if necessary.

[0189] In some embodiments, the length and area of ​​grassland damage can be calculated based on factors such as the speed and distance of the violating vehicle, and the violating vehicle can be punished based on comprehensive factors such as its location in the grassland, length of stay, and degree of grassland damage.

[0190] In some embodiments, human intrusion, abnormal fire-starting and other behaviors can be monitored, and alarms can be issued when abnormal events such as illegal fire-starting and abnormal smoke and fire in scenic spots are discovered, thereby protecting the safety of scenic spots, grasslands and nature reserves.

[0191] In some embodiments, algorithms such as crowd gathering and flow statistics are also supported, which can be used for counting the number of people in scenic spots and for scenic spot flow assessment to reduce damage to the scenic spot caused by too many people exceeding the scenic spot's tolerance range.

[0192] (VI) Mobile terminal linkage module.

[0193] In some embodiments, a mobile terminal linkage module is proposed, which can obtain data from the server so that the staff can view the details of the restricted area in real time through the mobile terminal.

[0194] In some embodiments, patrol personnel can be equipped with AR recorders, which can record the event handling process and have a positioning function. The event handler can be determined based on the patrol personnel's location, and the event handling efficiency can be improved by handling the event nearby.

[0195] As can be seen from the above, the embodiments of the present application can effectively monitor uncivilized behaviors in scenic restricted areas, grasslands, and nature reserves, timely discover and dissuade them, protect the environment of scenic areas, and maintain the safety of scenic areas. Record and accurately analyze the losses of scenic areas, drive away and stop illegal targets, and eliminate uncivilized tourist behaviors.

[0196] In addition, in addition to being applied to tourist attractions such as scenic spots and nature reserves, this solution can also be applied to other scenarios with restricted areas, such as traffic scenarios to prevent vehicles from entering driving restricted areas such as pedestrian walkways and non-motorized vehicle lanes; and security scenarios such as preventing tourists and pedestrians from entering dangerous areas, private land, restricted areas and other non-public areas.

[0197] In order to better implement the above method, the embodiment of the present application also provides a warning device, which can be integrated in an electronic device, and the electronic device can be a terminal, a server, etc. Among them, the terminal can be a mobile phone, a tablet computer, a smart Bluetooth device, a laptop, a personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers.

[0198] For example, in this embodiment, the method of the embodiment of the present application will be described in detail by taking the example that the warning device is specifically integrated into the server.

[0199] For example, Figure 3 As shown, the warning device may include a video unit 301, a recognition unit 302, and a warning unit 303, as follows:

[0200] (a) Video unit 301.

[0201] The video unit 301 is used to obtain an observation video, where the observation video includes a currently observed image of a target area, and the target area includes a restricted area and an open area.

[0202] In some embodiments, obtaining the observed video includes:

[0203] Obtain observation videos collected in the target area;

[0204] Display the images of the target area in the observation video;

[0205] In response to the area setting operation for the screen, a restricted area and an open area of ​​the target area are determined in the screen.

[0206] In some embodiments, the area setting operation includes a segmentation operation, and in response to the area setting operation on the picture, determining the restricted area and the open area of ​​the target area in the picture includes:

[0207] In response to a split operation on the screen, a split line corresponding to the split operation is displayed in the screen;

[0208] The target area in the image is segmented based on the segmentation line to obtain a restricted area and an open area.

[0209] In some embodiments, the area setting operation includes a restricted area drawing operation. In response to the area setting operation on the screen, determining a restricted area and an open area of ​​the target area in the screen includes:

[0210] In response to a restricted area drawing operation on the screen, displaying the drawn restricted area sub-area in the screen;

[0211] All restricted area sub-areas are combined to obtain the restricted area of ​​the target area;

[0212] Determine the open area, which is the area in the target area except the restricted area.

[0213] In some embodiments, determining a restricted area and an open area of ​​a target area in a picture includes:

[0214] A neural network is used to segment the image and determine the restricted area and open area of ​​the target area in the image.

[0215] (ii) Identification unit 302.

[0216] The recognition unit 302 is used to perform object recognition on the observation video to obtain a target object, where the target object is located in a restricted area in the observation video.

[0217] In some embodiments, after performing object recognition on the observed video and obtaining the target object, the method further includes:

[0218] Identify the target object and obtain the identity information of the target object;

[0219] Conduct warnings on target objects in the target area, including:

[0220] Based on the identity information of the target object, the target object is warned in the target area.

[0221] In some embodiments, the target object includes a target vehicle, the identity information includes license plate information, and performing identity recognition on the target object to obtain the identity information of the target object includes:

[0222] Locate the license plate position of the target vehicle in the observation video;

[0223] Text recognition is performed based on the license plate position to obtain the license plate information of the target vehicle.

[0224] In some embodiments, after performing object recognition on the observed video and obtaining the target object, the method further includes:

[0225] When displaying the observation video, an outer frame is added to the target object, and the outer frame is used to highlight the target object in the observation video.

[0226] In some embodiments, after performing object recognition on the observed video and obtaining the target object, the method further includes:

[0227] When the target object is located in the restricted area for the first time, the first entry time of the target object is recorded;

[0228] When the target object is not in the restricted area, the departure time of the target object is recorded;

[0229] Based on the first entry time and the exit time, the residence time of the target object is determined, and the residence time represents the time the target object stays in the restricted area.

[0230] In some embodiments, after performing object recognition on the observed video and obtaining the target object, the method further includes:

[0231] Obtain the target object's behavior information during the detention time;

[0232] Determine the target's violation based on behavioral information;

[0233] Conduct warnings on target objects in the target area, including:

[0234] Based on the target object's illegal behavior, the target object is warned in the target area.

[0235] In some embodiments, after determining the illegal behavior of the target object based on the behavior information, the method further includes:

[0236] The residence time, behavior information, and violation behavior of the target object are sent to the mobile terminal so that the mobile terminal can view the target object, as well as the residence time, behavior information, and violation behavior of the target object.

[0237] (iii) Warning unit 303.

[0238] The warning unit 303 is used to perform warning processing on the target object in the target heap area.

[0239] In some embodiments, performing warning processing on a target object in a target heap area includes:

[0240] A warning message is sent to a warning device, the warning device is installed in a target area, and the warning message carries a message for warning the target object, so that the warning device broadcasts the warning message in the target area.

[0241] In specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can refer to the previous method embodiments, which will not be repeated here.

[0242] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0243] As can be seen from the above, the warning device of this embodiment obtains the observation video by the video unit, and the observation video includes the current observation of the target area, and the target area includes the restricted area and the open area; the recognition unit performs object recognition on the observation video to obtain the target object, and the target object is located in the restricted area in the observation video; the warning unit performs warning processing on the target object in the target pile area. Therefore, the embodiment of the present application can improve the warning efficiency.

[0244] The embodiment of the present application also provides an electronic device, which can be a terminal, a server, etc. The terminal can be a mobile phone, a tablet computer, a smart Bluetooth device, a laptop, a personal computer, etc. The server can be a single server or a server cluster composed of multiple servers, etc.

[0245] In some embodiments, the warning device may also be integrated into multiple electronic devices. For example, the warning device may be integrated into multiple servers, and the warning method of the present application may be implemented by multiple servers.

[0246] In this embodiment, the electronic device of this embodiment is a server as an example for detailed description, for example, Figure 4 As shown, it shows a schematic diagram of the structure of the electronic device involved in the embodiment of the present application, specifically:

[0247] The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, an input module 404, and a communication module 405. Those skilled in the art will appreciate that Figure 4 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0248] The processor 401 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 402, and calling data stored in the memory 402, it performs various functions of the electronic device and processes data, thereby performing overall detection of the electronic device. In some embodiments, the processor 401 may include one or more processing cores; in some embodiments, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 401.

[0249] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0250] The electronic device also includes a power supply 403 for supplying power to various components. In some embodiments, the power supply 403 can be logically connected to the processor 401 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 403 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0251] The electronic device may further include an input module 404, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0252] The electronic device may further include a communication module 405. In some embodiments, the communication module 405 may include a wireless module. The electronic device may perform short-range wireless transmission through the wireless module of the communication module 405, thereby providing the user with wireless broadband Internet access. For example, the communication module 405 may be used to help the user send and receive emails, browse web pages, and access streaming media.

[0253] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 401 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 402 according to the following instructions, and the processor 401 will run the application programs stored in the memory 402, thereby realizing various functions, as follows:

[0254] Obtaining observation video, the observation video includes the currently observed images of the target area, and the target area includes the restricted area and the open area;

[0255] Performing object recognition on the observation video to obtain the target object, where the target object is located in the restricted area in the observation video;

[0256] Provide warning to target objects in the target area.

[0257] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0258] It can be seen from the above that the embodiments of the present application can reduce labor costs and thus improve warning efficiency.

[0259] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0260] To this end, an embodiment of the present application provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any of the warning methods provided in the embodiments of the present application. For example, the instructions can execute the following steps:

[0261] Obtaining observation video, the observation video includes the currently observed images of the target area, and the target area includes the restricted area and the open area;

[0262] Performing object recognition on the observation video to obtain the target object, where the target object is located in the restricted area in the observation video;

[0263] Provide warning to target objects in the target area.

[0264] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0265] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in various optional implementations of the scenic area warning aspect or the restricted area security aspect provided in the above embodiments.

[0266] Since the instructions stored in the storage medium can execute the steps in any warning method provided in the embodiments of the present application, the beneficial effects that can be achieved by any warning method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0267] The above is a detailed introduction to a warning method, device, electronic device and computer-readable storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A warning method, It is characterized in that include: Acquire an observation video, wherein the observation video includes a currently observed image of a target area, wherein the target area includes a restricted area and an open area; Performing object recognition on the observation video to obtain a target object, wherein the target object is located in the restricted area in the observation video; Performing warning processing on the target object in the target heap area.

2. The warning method according to claim 1, It is characterized in that The obtaining of the observation video comprises: Obtain observation videos collected in the target area; Displaying a picture of the target area in the observation video; In response to an area setting operation for the picture, a restricted area and an open area of ​​the target area are determined in the picture.

3. The warning method according to claim 2, It is characterized in that The area setting operation includes a segmentation operation, and in response to the area setting operation on the picture, determining the restricted area and the open area of ​​the target area in the picture includes: In response to a split operation on the screen, displaying a split line corresponding to the split operation in the screen; The target area in the picture is segmented based on the segmentation line to obtain a restricted area and an open area.

4. The warning method according to claim 2, It is characterized in that The area setting operation includes a restricted area drawing operation, and in response to the area setting operation on the picture, determining the restricted area and the open area of ​​the target area in the picture includes: In response to a restricted area drawing operation on the picture, displaying a drawn restricted area sub-area in the picture; Performing a union process on all restricted area sub-areas to obtain a restricted area of ​​the target area; An open area is determined, where the open area is an area in the target area except the restricted area.

5. The warning method according to claim 2, It is characterized in that Determining the restricted area and the open area of ​​the target area in the picture includes: A neural network is used to perform region segmentation processing on the image, and a restricted area and an open area of ​​the target area are determined in the image.

6. The warning method according to claim 1, It is characterized in that The step of performing warning processing on the target object in the target heap area includes: A warning message is sent to a warning device, the warning device is installed in the target area, the warning message carries a message for warning the target object, so that the warning device broadcasts the warning message in the target area.

7. The warning method according to claim 1, It is characterized in that After performing object recognition on the observed video to obtain the target object, the method further includes: Performing identity recognition on the target object to obtain identity information of the target object; The performing warning processing on the target object in the target heap area includes: Perform warning processing on the target object in the target heap area based on the identity information of the target object.

8. The warning method according to claim 7, It is characterized in that The target object includes a target vehicle, the identity information includes license plate information, and the identification of the target object to obtain the identity information of the target object includes: Locating the license plate position of the target vehicle in the observation video; Text recognition is performed based on the license plate position to obtain the license plate information of the target vehicle.

9. The warning method according to claim 1, It is characterized in that After performing object recognition on the observed video to obtain the target object, the method further includes: When displaying the observation video, an outer frame is added to the target object, and the outer frame is used to highlight the target object in the observation video.

10. The warning method according to claim 1, It is characterized in that After performing object recognition on the observed video to obtain the target object, the method further includes: When the target object is located in the restricted area for the first time, recording the first entry time of the target object; When the target object is not located in the restricted area, recording the departure time of the target object; The retention time of the target object is determined based on the first entry time and the exit time, where the retention time indicates the time the target object stays in the restricted area.

11. The warning method according to claim 10, It is characterized in that After performing object recognition on the observed video to obtain the target object, the method further includes: Obtaining behavior information of the target object during the retention time; Determining the illegal behavior of the target object based on the behavior information; The performing warning processing on the target object in the target heap area includes: Based on the illegal behavior of the target object, a warning process is performed on the target object in the target heap area.

12. The warning method according to claim 11, It is characterized in that After determining the illegal behavior of the target object based on the behavior information, the method further includes: The residence time, behavior information and violation of the target object are sent to a mobile terminal so that the mobile terminal can view the target object, the residence time, behavior information and violation of the target object.

13. A warning device, It is characterized in that include: A video unit, used to obtain an observation video, wherein the observation video includes a currently observed picture of a target area, wherein the target area includes a restricted area and an open area; an identification unit, configured to perform object identification on the observation video to obtain a target object, wherein the target object is located in the restricted area in the observation video; The warning unit is used to perform warning processing on the target object in the target heap area.

14. An electronic device, It is characterized in that It comprises a processor and a memory, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to execute the steps in the warning method according to any one of claims 1 to 12.

15. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the warning method according to any one of claims 1 to 12.

Images