An automatic regional equipment linkage method based on video intelligent object detection technology

Abstract

The present invention discloses an automatic regional equipment linkage method based on video intelligent object detection technology, comprising the following steps: training personnel recognition model; recognition area pre-division; deployment of edge computing hardware; intelligent equipment linkage; long-term static portrait elimination; regional anti-tremor processing. The present invention realizes intelligent equipment linkage in the classroom, which can greatly improve the user experience of the smart classroom. When there are people, turn on the lights and air conditioners, and when there are no people, turn off the lights and air conditioners. The equipment is switched on and off by area, so as to achieve refined management of the equipment and energy saving and emission reduction. In the smart classroom scene, the management personnel can know the distribution of students' seats during class through the heat map, so as to analyze which courses can attract students and which courses cannot. The use of the present invention can know the number of people in the classroom at all times, reduce the phenomenon of skipping classes, and effectively reduce the safety hazards caused by students leaving the classroom. The present invention can be applied not only to smart classrooms, but also to other similar industries such as smart homes.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent control equipment linkage, and in particular to an automatic regional equipment linkage method based on video intelligent object detection technology. Background Art

[0002] At present, the technologies for linking smart control devices such as smart homes and smart classrooms all have a key requirement, which is to identify the distribution of people in the space, so as to control the devices by area according to the distribution of people, turning on the devices in areas with people and turning off the devices in areas without people. This can not only enable people to control the devices without feeling, but also reduce energy waste.

[0003] Traditional personnel distribution recognition technology mainly uses infrared probes for personnel recognition. This technology requires the pre-installation and deployment of multiple infrared probes in the space. When the infrared probe detects that there are people in the area, it sends a signal to the central control host to achieve linkage.

[0004] The main problems of using infrared probes for personnel identification are as follows:

[0005] 1. The infrared probe needs to be constructed and arranged in advance, which will cause certain construction and maintenance costs and affect the aesthetics of the space;

[0006] 2. The infrared probe recognition accuracy is not high, and it cannot accurately locate people and accurately identify the number of people;

[0007] 3. The recognition areas of multiple infrared probes are bound to overlap and miss each other;

[0008] 4. The infrared probe cannot identify stationary personnel;

[0009] 5. The detection area of ​​the infrared probe cannot be changed after the infrared probe is installed.

[0010] Due to various problems with infrared personnel recognition technology, this technology has a poor experience in the field of regional personnel distribution recognition and is difficult to promote on a large scale. Therefore, there is an urgent need to invent new technologies to achieve accurate recognition of personnel spatial distribution. Summary of the invention

[0011] In view of this, in order to solve the above-mentioned problems in the prior art, the present invention proposes an automatic regional equipment linkage method based on video intelligent object detection technology, which adopts edge computing hardware and uses the latest deep learning object recognition technology to identify the distribution of people in video images, and then analyzes the distribution of people in the area based on the pre-defined polygonal area, and realizes regional equipment linkage based on this.

[0012] The present invention solves the above problems by the following technical means:

[0013] An automatic regional device linkage method based on video intelligent object detection technology comprises the following steps:

[0014] Build a deep neural network model, obtain a large number of labeled learning sample images, use the learning sample images to train the deep neural network model, and use the trained deep neural network model as a person recognition model;

[0015] According to the situation of the video recognition site and the location of the equipment installation and layout, the recognition site is divided into multiple areas;

[0016] Deploy edge computing hardware. The video from each camera is processed separately using edge computing hardware.

[0017] The edge computing hardware uses the personnel recognition model to analyze the video stream frame by frame, analyzes the position of the head portrait in each frame, and then calculates the coordinates of the center point of the head portrait. After calculating the recognition area where the center point is located, the distribution information of the people in the video is obtained; after the regional distribution of the people is obtained, the devices in the area are linked;

[0018] Analyze whether there are people standing still for a long time at the monitored scene, and remove those people standing still for a long time;

[0019] For regional anti-vibration processing, the intersection of two adjacent areas is used as the regional handover area. The equipment linkage method of the regional handover area is specially processed. The principle of equipment linkage processing in the handover area is to reduce the area switching of personnel, thereby reducing the frequent switching of equipment.

[0020] Furthermore, the specific method for training the deep neural network model is:

[0021] Obtain a large number of labeled learning sample images, and divide the learning sample images into training sets, validation sets, and test sets;

[0022] Put the learning sample images of the training set into the deep neural network model to train the deep neural network model;

[0023] Put the learning sample images of the validation set into the deep neural network model, adjust the network parameters of the deep neural network model, and improve the recognition accuracy of the deep neural network model;

[0024] Put the learning sample images of the test set into the deep neural network model to predict the classification results.

[0025] Furthermore, the specific method of removing long-term static portraits is:

[0026] Set the static portrait recognition time period and related parameters;

[0027] Identify the position of the portrait in the video and superimpose it frame by frame;

[0028] When the static recognition period ends, a dot matrix is ​​obtained;

[0029] Gaussian blur the dot map to get the heat map;

[0030] In the area above the threshold in the heat map, the portrait is removed.

[0031] Furthermore, the specific method for setting the static portrait recognition time period is:

[0032] The time period for stationary person recognition is selected from the time period when the entire scene can be clearly seen in the scene, including the time in the morning and evening when the lighting is sufficient; it can be specified through the setting interface; the stationary recognition time period is adjusted according to the business situation, and the time when there are too many people is not suitable for the recognition of stationary persons, including the time when students are in class; when performing stationary person recognition, specify the associated class schedule and intelligently adjust the stationary portrait recognition time.

[0033] Furthermore, the specific method of identifying the position of the portrait in the video and superimposing it frame by frame is:

[0034] While performing deep learning personnel position recognition operations, a special location will be allocated in the memory to record the location where the portrait appears in each frame. When the center of the portrait appears at a certain coordinate, the array element corresponding to the coordinate is increased by 1, thereby accumulating the number of people appearing at each coordinate.

[0035] Furthermore, when the static recognition period ends, the specific method of obtaining the dot matrix is:

[0036] When the preset static recognition time period ends, a coordinate dot matrix will be obtained, which appears in the form of an array; each pixel in the image corresponds to an array element, and the value of the element is the number of times the portrait midpoint appears in the pixel. When the element value is large, it means that the portrait terminal appears frequently in the pixel.

[0037] Furthermore, the specific method of Gaussian blurring the dot map to obtain the heat map is:

[0038] Use Gaussian filtering on the dot map to connect the densely scattered points into a dense area to obtain a heat map;

[0039] Gaussian filter formula, where x and y are the horizontal and vertical axis distances from the calculation point, σ is the smoothing coefficient, and e is a natural constant.

[0040]

[0041] Furthermore, in the area exceeding the threshold in the heat map, the specific method of removing the portrait is:

[0042] After obtaining the regional thermal image, the density-based clustering algorithm is used to cluster the adjacent pixels with a high number of portraits into one category, thereby obtaining several separate areas with densely populated people. Then, each area is analyzed one by one. When the portrait frequency index of a region exceeds the threshold, the region is defined as a static portrait area, and the portraits appearing in this area are regarded as static portraits and removed.

[0043] The static area recognition program will continue to run while the person recognition program is running, thereby continuously updating the static portrait recognition area. When it is found that the static portrait is removed, the static portrait rejection area is also removed accordingly.

[0044] Furthermore, the specific method of regional anti-tremor processing is:

[0045] Set the portrait recognition areas to intersect;

[0046] Automatically identify the intersection between areas;

[0047] Special treatment of the avatar in the handover area. The principle of special treatment of the handover area is to reduce the area switching of personnel.

[0048] Furthermore, the portrait recognition areas are set to intersect. The specific method is: for areas where switching back and forth between adjacent areas is easy to occur, the areas are set to have intersections as the intersection areas; and when setting the areas, it is necessary to avoid the situation where three areas intersect, areas contain and multiple intersection areas;

[0049] The specific method of automatically identifying the intersection between areas is: when reading the data of the identification area, it will automatically use the method of computational geometry to identify the intersection between the areas and record its coordinate position. When identifying the personnel area, the intersection area will be specially processed; when identifying the incorrect setting of the intersection area, it will give corresponding prompts;

[0050] The specific method of special treatment for the avatar in the handover area is: the handover area can only belong to one of its two related areas; when judging the area to which the handover area belongs, compare the two adjacent areas related to the handover area, calculate the time when a person last appeared there, and give priority to selecting the area where a person appeared latest as the area to which the handover area belongs.

[0051] Compared with the prior art, the beneficial effects of the present invention include at least:

[0052] 1. Currently, most classrooms are equipped with cameras. The present invention can make use of old cameras. To use the present invention, you only need to install an edge computing device and coordinate it with the linked hardware. The edge computing device can read the video stream from the camera, analyze the distribution of personnel, and then coordinate the devices. Moreover, the edge computing device can be remotely configured and remotely maintained.

[0053] 2. Realizing the linkage of intelligent devices in the classroom can greatly improve the user experience of the intelligent classroom. When someone is there, turn on the lights and air conditioner, and when no one is there, turn off the lights and air conditioner, and can switch the equipment by area. In some schools, classrooms are often unattended during the evening self-study period, and all the equipment is still turned on, resulting in a huge waste of energy. After using the present invention, it is possible to achieve refined management of equipment and save energy and reduce emissions.

[0054] 3. The present invention can output a heat map of personnel distribution, which can also be used as a reference for managers. In the smart classroom scenario, managers can know the distribution of students' seats during class through the heat map, so as to analyze which courses can attract students and which courses cannot.

[0055] 4. The present invention can also be used in conjunction with an attendance system. The attendance system can only count the number of people in the classroom at a certain time. However, the present invention can always know the number of people in the classroom and report when the number of people in the classroom is less than a certain number, thereby more effectively managing classroom discipline and reducing the phenomenon of skipping classes. It can also effectively reduce the safety hazards caused by students leaving the classroom.

[0056] 5. The present invention can be used to know the position of people in the classroom, and can be used to judge the social distance between people. At present, there is still a risk of an outbreak of the new crown epidemic, and people must maintain a certain social distance at certain times. The present invention can identify the position of people in space, thereby obtaining the distance between people, and issue an alarm when the distance between two people is less than a certain threshold.

[0057] 6. The present invention can be applied not only to smart classrooms, but also to other similar industries such as smart homes. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0059] Figure 1 It is a flow chart of the automatic regional equipment linkage method based on video intelligent object detection technology of the present invention;

[0060] Figure 2 This is a flow chart of long-term static portrait elimination of the present invention;

[0061] Figure 3 It is a result diagram of the thermal map generation of the present invention;

[0062] Figure 4 It is a flow chart of the density-based clustering algorithm of the present invention;

[0063] Figure 5 It is a schematic diagram of the inventor constantly switching between adjacent areas due to slight movements;

[0064] Figure 6 It is a specific processing flow chart of regional anti-tremor processing of the present invention;

[0065] Figure 7 It is a schematic diagram of correct and incorrect setting of the handover area of ​​the present invention. DETAILED DESCRIPTION

[0066] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be pointed out that the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0067] Since cameras in public spaces are now very popular, it provides a prerequisite for obtaining accurate information on the spatial distribution of people through video image analysis. The present invention adopts edge computing hardware and the latest deep learning object recognition technology to identify the location of people in video images, and then analyzes the distribution of people in the area based on pre-defined polygonal areas, and uses this as a basis to realize regional equipment linkage.

[0068] like Figure 1 As shown, the present invention provides an automatic regional device linkage method based on video intelligent object detection technology, comprising the following steps:

[0069] S1. Training Person Recognition Model

[0070] Object detection algorithm is an important application of the currently popular deep learning intelligent image recognition technology. This technology needs to detect specific objects in icons, which includes two functions:

[0071] Localization: The detector needs to give the location of the object in the image;

[0072] Classification: The detector needs to give the category to which the object belongs.

[0073] Training a person recognition model first requires a large number of labeled learning samples. The present invention collects a large number of pictures containing people, including pictures of people in various backgrounds, different lighting, multiple scenes, multiple angles, and different densities. The people in the pictures include various ages, genders, races, skin colors, postures, and expressions. The collected pictures are about 10g. Some of these pictures are based on the position of the head portrait marked, and some need to be manually marked. After completing the picture marking, the pictures are divided into training set, verification set, and test set, ready for training.

[0074] At present, deep learning image object detection technology has made great progress, with significant improvements in accuracy and speed, and even outperforms human recognition on some data sets. Object detection algorithms are usually divided into two-stage algorithms and one-stage algorithms, which refer to whether the location and classification of objects are completed in two stages or in one stage. The deep neural network of the algorithm used in the present invention is divided into three parts, namely: backbone, neck, and top of the head. The three parts are used for feature extraction, feature fusion, and result output respectively.

[0075] After obtaining the training set and building the deep neural network, the model training is carried out using cloud computing. The training takes about 2 days and runs more than 5,000 cycles. The training is completed after the recognition accuracy of the validation set has not been significantly improved. After obtaining the network parameters of the deep neural network, the image object detection model can be inferred.

[0076] S2. Identify area pre-partitioning

[0077] Pre-zoning of the recognition area means dividing the recognition site into multiple areas according to the situation of the video recognition site and the installation and layout of the lighting, air conditioning and other equipment on site. When the video detection algorithm recognizes that there is a person in the area, the equipment in the area is turned on. When the person leaves for a certain period of time, the equipment in the area is turned off.

[0078] The area division configuration of the present invention is implemented using a B / S architecture. The system captures a screenshot of the video on the web page, and the operator only needs to click on the screen window to complete the polygonal area division. In addition, the area division data can be synchronized, copied, and backed up. Different linkage areas can be set according to different time periods, and the sensitivity of area recognition can be specified.

[0079] Compared with the method of fixing the personnel recognition area using infrared personnel recognition technology, the area division method of the present invention is more sophisticated, more flexible and more accurate.

[0080] S3. Deploy edge computing hardware

[0081] Since deep learning video detection technology is a computationally intensive program, the hardware devices running deep learning programs are always under high load, so special hardware devices are generally required for computing. There are two main ways to deploy deep learning computing systems: centralized deployment and edge computing deployment. Centralized deployment refers to the use of a central server, where all videos are aggregated to the central server for processing and analysis. In practice, centralized deployment will cause problems such as excessive network burden, inflexible hardware deployment, and high deployment costs.

[0082] The present invention uses edge computing to deploy hardware. The advantage of this method is that the video of each camera is processed separately using edge computing hardware, and then only the processing results are uploaded to the central server and control device, without the need to transmit a large amount of video. The advantage of this method is that it saves traffic and has flexible deployment.

[0083] S4, Smart device linkage

[0084] The edge hardware uses a deep learning network to analyze the video stream frame by frame, analyzes the position of the head portrait in each frame, and then calculates the coordinates of the center point of the head portrait. After calculating the recognition area where the center point is located, the distribution information of the people in the video is obtained.

[0085] After the distribution of personnel in different areas is obtained, the devices in the area are linked. For example, lights are turned on when there are people, and lights are turned off when there are no people. The linkage information is constructed using a private protocol and sent using the TCP / IP protocol.

[0086] S5. Long-term static portrait removal

[0087] Intelligent personnel recognition systems need to be used in various scenarios in reality. In the scenarios where personnel recognition is performed, there may be portraits or human models, which will cause long-term false alarms and seriously affect the accuracy of recognition. Therefore, it is necessary to eliminate long-term static portraits to improve the accuracy of the system and enhance the user experience of the system.

[0088] Traditional liveness recognition technology is for a single person. These liveness recognition technologies generally require the person being analyzed to be in front of the camera alone and make different facial expressions according to the instructions to prove that it is not a picture or screen video. The scenario for intelligent person recognition in the present invention is to use a surveillance camera to recognize multiple portraits at the same time, which is far different from the traditional liveness recognition usage scenario and it is difficult to directly apply the relevant technology.

[0089] The long-term static portrait recognition technology used in the present invention is to analyze whether there are people who have not moved for a long time at the monitored scene, that is, the position of the person has not changed for a long time. At this time, the system regards it as a portrait of the person at the scene and removes it during analysis. The system will continuously analyze the situation at the scene, and automatically restore the removed area when there is no portrait at the removed position.

[0090] The specific processing flow of long-term static portrait removal is as follows Figure 2 As shown, the following steps are included:

[0091] S51. Setting the static portrait recognition time period and related parameters

[0092] The time period for stationary person recognition is generally selected from the time period when the entire scene can be clearly seen in the scene, such as the time in the morning and evening when the lights are sufficient. In the present invention, this can be specified through the setting interface. The stationary recognition time period may also be adjusted according to business conditions. For example, a time when there are too many people is not suitable for stationary person recognition, such as the time when students are in class. In the present invention, when performing stationary person recognition, the associated course schedule can be specified to intelligently adjust the stationary portrait recognition time.

[0093] S52, identifying the position of the portrait in the video and superimposing it frame by frame

[0094] While performing deep learning personnel position recognition operations, the system of the present invention will open a special location in the memory to record the position where the portrait appears in each frame. When the center of the portrait appears at a certain coordinate, the array element corresponding to the coordinate is increased by 1, thereby accumulating the number of people appearing at each coordinate.

[0095] S53, the static recognition time period ends, and a dot matrix is ​​obtained

[0096] When the preset static recognition time period ends, a coordinate dot matrix will be obtained, which appears in the form of an array. Each pixel in the image corresponds to an array element, and the value of the element is the number of times the portrait midpoint appears in the pixel. When the element value is large, it means that the portrait terminal appears frequently in the pixel.

[0097] S54. Perform Gaussian blur on the dot map to obtain a heat map

[0098] Theoretically, we can remove pixels that appear too frequently in the center of the portrait based on the dot map to achieve static portrait recognition. In practical applications, although the position of the portrait image will not change, the surrounding environment will change, such as the lighting of the scene will change, and the portrait will be partially blocked. Due to the complexity and sensitivity of the deep learning network, the position of the recognized portrait will change slightly. Therefore, although the position of the portrait image has not moved, the center of the recognized portrait will move and become multiple scattered points in the dot map.

[0099] The present invention uses Gaussian filtering to process the dot map, so that densely scattered points are connected into a dense area to obtain a heat map.

[0100] Gaussian filter formula, where x and y are the horizontal and vertical axis distances from the calculation point, σ is the smoothing coefficient, and e is a natural constant.

[0101]

[0102] The heat map generation results are as follows Figure 3 shown.

[0103] S55. Eliminate the portrait in the area exceeding the threshold in the heat map

[0104] After obtaining the regional thermal image, the present invention uses a density-based clustering algorithm (such as Figure 4 As shown in the figure, the adjacent pixels with more portraits are grouped together, thus obtaining several separate areas with densely populated people. Then the areas are analyzed one by one. When the portrait frequency index of an area exceeds the threshold, the area is defined as a static portrait area, and the portraits in the area are regarded as static portraits and removed.

[0105] The static area recognition program will continue to run while the person recognition program is running, thereby continuously updating the static portrait recognition area. When a static portrait is found to be removed, the static portrait rejection area is also removed accordingly.

[0106] S6, Regional Anti-Shake Processing

[0107] When analyzing personnel areas and linking regional equipment, if personnel keep switching between adjacent areas due to slight movements (such as Figure 5 As shown in the figure, the devices in two adjacent areas will be constantly switched on and off, which will seriously affect the user experience and shorten the service life of the devices.

[0108] The present invention uses a method for identifying the handover area, and regards the intersection of two adjacent areas as the area handover area. The equipment linkage mode of the area handover area is specially processed. The principle of equipment linkage processing in the handover area is to minimize the area switching of personnel, thereby reducing the frequent switching of equipment.

[0109] The specific process of regional anti-tremor treatment is as follows Figure 6 As shown, the following steps are included:

[0110] S61. Set the portrait recognition area to intersect

[0111] When setting the person recognition area on the video screenshot by clicking on the web page, follow some rules. For areas where it is easy to switch back and forth between adjacent areas, it is necessary to set the intersection between the areas as the handover area. And when setting the area, it is necessary to avoid the situation where three areas intersect, areas contain, and multiple intersecting areas, such as Figure 7 shown.

[0112] S62, automatic identification of the intersection between areas

[0113] When the system of the present invention reads the data of the identification area, it will automatically use the method of computational geometry to identify the intersection area between the areas and record its coordinate position. When identifying the personnel area, the intersection area is specially processed. The system will also identify the situation where the intersection area is incorrectly set (i.e. the intersection of the three areas mentioned above, etc.) and give corresponding prompts.

[0114] S63. Special treatment of the head portrait in the handover area

[0115] The special processing principle of the handover area is to minimize the area switching of personnel. The specific method is: the handover area can only belong to one of the two related areas. When judging the area to which the handover area belongs, the system compares the two adjacent areas related to the handover area, calculates the time when the personnel last appeared in them, and the system gives priority to the area where the personnel appeared the latest as the area to which the handover area belongs.

[0116] The present invention can be used in multiple industries. The following example uses it in a smart classroom, and other industries are similar.

[0117] At present, multimedia classrooms generally have installed and deployed several hosts:

[0118] 1. Electronic control host used to control classroom electrical appliances (lighting, air conditioning, etc.);

[0119] 2. Central control host used to control classroom multimedia (computers, projectors, screens, etc.);

[0120] 3. Camera used to display the situation in the classroom.

[0121] The control and monitoring facilities in the multimedia classroom are quite complete, but they lack connection with each other. The electric control and central control in the classroom operate independently, and the equipment in the classroom is turned on and off manually according to various protocols; the camera can be connected to the central control room and monitored by specialized personnel. Although the multimedia classroom can conveniently centrally control and monitor the equipment, there are still some problems:

[0122] 1. The lights cannot be automatically controlled by area, so that the corresponding lights are turned on when there are people in the area and turned off when there are no people;

[0123] 2. Various computers, projectors, screens and other multimedia equipment cannot be automatically turned off according to the number of people in the room. Once people forget to turn off the equipment after leaving, it will cause energy waste, shorten the life of the equipment, and cause certain safety issues;

[0124] 3. If you want to know the number and distribution of people in the classroom, you can only review it through manual monitoring, which wastes human resources and makes it difficult to achieve real-time reporting.

[0125] In view of the current status of multimedia classrooms, the present invention uses edge computing hardware installed with artificial intelligence vision programs. The edge computing hardware can read the video information of the camera through the network, analyze it in real time, and analyze the distribution of people in the video screen. At the same time, the edge computing hardware can also connect to the existing electronic control and central control host of the multimedia classroom, send instructions to control various devices in the classroom through the electronic control or central control, thereby realizing intelligent control of classroom equipment.

[0126] The present invention has great benefits in the practice of intelligent multimedia classrooms:

[0127] 1. The present invention can fully utilize the old equipment; the original host and equipment in the classroom do not need to be modified. As long as the original electric control, central control, and camera are connected to the edge computing hardware through the network, these original facilities in the classroom can be fully utilized to realize the intelligent classroom;

[0128] 2. The present invention is easy to install; using the present invention, only one edge computing device needs to be deployed to realize the intelligentization of multimedia classrooms; edge computing has a small size, low power consumption, and can be remotely maintained;

[0129] 3. The present invention can solve the problem of intelligently controlling devices by region; the present invention can be used to determine the light switch, brightness or air conditioning temperature according to the regional distribution of personnel and the number of people;

[0130] 4. At present, the central control system of multimedia classrooms has the problem of how to deal with the situation when the teacher forgets to turn off the system after class. Usually, the problem is solved by timed forced shutdown or inspection by maintenance personnel of electronic teaching equipment. Timed shutdown cannot solve the problem of teachers staying in class, and inspection by maintenance personnel increases costs. The present invention can analyze the real-time total number of people in the classroom. When no one is in the classroom for a period of time, the central control system can be linked to automatically shut down the multimedia equipment, solving a major problem in the central control industry of multimedia classrooms.

[0131] 5. The present invention can automatically analyze the distribution of people in the classroom; the present invention can automatically analyze the distribution of people in the classroom through video, and can automatically output the result table and heat map; it has important reference value for the school's learning situation analysis and classroom management;

[0132] 6. Currently, most classrooms are already equipped with cameras. The present invention can make use of old cameras. To use the present invention, you only need to install an edge computing device and coordinate it with the linked hardware. The edge computing device can read the video stream from the camera, analyze the distribution of personnel, and then coordinate the devices. Moreover, the edge computing device can be remotely configured and remotely maintained;

[0133] 7. Realizing the linkage of smart devices in the classroom can greatly improve the user experience of the smart classroom. When there are people, turn on the lights and air conditioners, and when there are no people, turn off the lights and air conditioners, and can switch the equipment by area. In some schools, classrooms are often unattended during the evening self-study period, and all the equipment is still turned on, resulting in a huge waste of energy. After using the present invention, it is possible to achieve refined management of equipment and save energy and reduce emissions;

[0134] 8. The present invention can output a heat map of personnel distribution, which can also be used as a reference for managers. In the smart classroom scenario, managers can know the distribution of students' seats during class through the heat map, so as to analyze which courses can attract students and which courses cannot.

[0135] 9. The present invention can also be used in conjunction with an attendance system; an attendance system can only count the number of people in a classroom at a certain time; however, the present invention can know the number of people in the classroom at all times, and report when the number of people in the classroom is less than a certain number, thereby more effectively managing classroom discipline and reducing truancy; it can also effectively reduce the potential safety hazards caused by students leaving the classroom;

[0136] 10. The present invention can be used to know the position of people in the classroom, and can be used to judge the social distance between people. At present, there is still a risk of an outbreak of the new crown epidemic, and people must maintain a certain social distance at certain times; the present invention can identify the position of people in space, thereby obtaining the distance between people, and when the distance between two people is less than a certain threshold, an alarm is issued.

[0137] In addition to being used in multimedia smart classroom scenarios, the present invention is also applicable to many other scenarios, such as:

[0138] 1. Office space: Analyze the movement of personnel and overtime work by analyzing the distribution of personnel in the office space;

[0139] 2. Smart home: automatically analyzes whether someone breaks in after the owner leaves, suitable for home security;

[0140] 3. Public places: By analyzing the distribution of people in public places, it can be found out whether there are crowds of people.

[0141] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. An automatic regional equipment linkage method based on video intelligent object detection technology, characterized in that: The steps include: Build a deep neural network model, obtain a large number of labeled learning sample images, use the learning sample images to train the deep neural network model, and use the trained deep neural network model as a person recognition model; According to the situation of the video recognition site and the location of the equipment installation and layout, the recognition site is divided into multiple areas; Deploy edge computing hardware. The video from each camera is processed separately using edge computing hardware. The edge computing hardware uses the personnel recognition model to analyze the video stream frame by frame, analyzes the position of the head portrait in each frame, and then calculates the coordinates of the center point of the head portrait. After calculating the recognition area where the center point is located, the distribution information of the people in the video is obtained; after the regional distribution of the people is obtained, the devices in the area are linked; Analyze whether there are people standing still for a long time at the monitored scene, and remove those people standing still for a long time; Regional anti-vibration processing, the intersection of two adjacent areas is regarded as the regional handover area, and the equipment linkage mode of the regional handover area is specially processed. The principle of equipment linkage processing in the handover area is to reduce the area switching of personnel, thereby reducing the frequent switching of equipment; The specific method of special treatment for the avatar in the handover area is: the handover area can only belong to one of its two related areas; when judging the area to which the handover area belongs, compare the two adjacent areas related to the handover area, calculate the time when a person last appeared there, and give priority to selecting the area where a person appeared latest as the area to which the handover area belongs.

2. The automatic regional equipment linkage method based on video intelligent object detection technology according to claim 1 is characterized in that: The specific method for training the deep neural network model is: Obtain a large number of labeled learning sample images, and divide the learning sample images into training sets, validation sets, and test sets; Put the learning sample images of the training set into the deep neural network model to train the deep neural network model; Put the learning sample images of the validation set into the deep neural network model, adjust the network parameters of the deep neural network model, and improve the recognition accuracy of the deep neural network model; Put the learning sample images of the test set into the deep neural network model to predict the classification results.

3. The automatic regional equipment linkage method based on video intelligent object detection technology according to claim 1 is characterized in that: The specific method of long-term static portrait removal is: Set the static portrait recognition time period and related parameters; Identify the position of the portrait in the video and superimpose it frame by frame; The static recognition period ends and a dot matrix is ​​obtained; Gaussian blur the dot map to get the heat map; In the area above the threshold in the heat map, the portrait is removed.

4. The automatic regional equipment linkage method based on video intelligent object detection technology according to claim 3 is characterized in that: The specific method to set the static portrait recognition time period is: The time period for stationary person recognition is selected from the time period when the entire scene can be clearly seen in the scene, including the time in the morning and evening when the lighting is sufficient; it can be specified through the setting interface; the stationary recognition time period is adjusted according to the business situation, and the time when there are too many people is not suitable for stationary person recognition, including the time when students are in class; when performing stationary person recognition, specify the associated class schedule and intelligently adjust the stationary portrait recognition time.

5. The automatic regional equipment linkage method based on video intelligent object detection technology according to claim 3 is characterized in that: Identify the position of the portrait in the video and overlay it frame by frame. The specific method is: While performing deep learning personnel position recognition operations, a special location will be allocated in the memory to record the location where the portrait appears in each frame. When the center of the portrait appears at a certain coordinate, the array element corresponding to the coordinate is increased by 1, thereby accumulating the number of people appearing at each coordinate.

6. The automatic regional equipment linkage method based on video intelligent object detection technology according to claim 3 is characterized in that: When the static recognition period ends, the specific method to obtain the dot matrix is: When the preset static recognition time period ends, a coordinate dot matrix will be obtained, which appears in the form of an array; each pixel in the image corresponds to an array element, and the value of the element is the number of times the portrait midpoint appears in the pixel. When the element value is large, it means that the portrait terminal appears frequently in the pixel.

7. The automatic regional equipment linkage method based on video intelligent object detection technology according to claim 3 is characterized in that: The specific method of Gaussian blurring the dot map to obtain the heat map is: Use Gaussian filtering on the dot map to connect the densely scattered points into a dense area to obtain a heat map; Gaussian filter formula, where x and y are the horizontal and vertical axis distances from the calculation point, σ is the smoothing coefficient, and e is a natural constant.

8. The automatic regional equipment linkage method based on video intelligent object detection technology according to claim 3 is characterized in that: In the area exceeding the threshold in the heat map, the specific method of removing the portrait is: After obtaining the regional thermal image, the density-based clustering algorithm is used to cluster the adjacent pixels with a high number of portraits into one category, thereby obtaining several separate areas with densely populated people. Then, each area is analyzed one by one. When the portrait frequency index of a region exceeds the threshold, the region is defined as a static portrait area, and the portraits appearing in this area are regarded as static portraits and removed. The static area recognition program will continue to run while the person recognition program is running, thereby continuously updating the static portrait recognition area. When it is found that the static portrait is removed, the static portrait rejection area is also removed accordingly.

9. The automatic regional equipment linkage method based on video intelligent object detection technology according to claim 1 is characterized in that: The specific method of regional anti-tremor treatment is: Set the portrait recognition areas to intersect; Automatically identify the intersection between areas; Special treatment of the avatar in the handover area. The principle of special treatment of the handover area is to reduce the area switching of personnel.

10. The automatic regional equipment linkage method based on video intelligent object detection technology according to claim 9 is characterized in that: Set the portrait recognition area to make it intersect. The specific method is: for areas where switching back and forth between adjacent areas is easy to occur, set the areas to have intersections as the intersection areas; and when setting the areas, pay attention to avoid the situation where three areas intersect, areas are included, and multiple intersection areas; The specific method of automatically identifying the intersection of areas is: when reading the data of the identification area, it will automatically use the computational geometry method to identify the intersection of the areas and record its coordinate position. When identifying the personnel area, the intersection area will be specially processed; when it is identified that the intersection area is set incorrectly, corresponding prompts will be given.

Images