Seat detection and recognition system based on deep learning

Abstract

The application discloses a seat detection and recognition system based on deep learning, which comprises an image acquisition and transmission module, a seat detection module and a seat recognition module.The image acquisition and transmission module is used for transmitting image information collected by a camera to a server.After the server processes real-time video images of seats collected by the camera, part of the image frames is intercepted and transmitted to the seat detection module as image features after anti-distortion processing.The seat detection module detects the image features by using a pre-trained deep learning detection model and segments each seat.The seat recognition module uses a pre-trained ResNet101 network to recognize the current state of each segmented seat in real time.The user interface module is used for visually displaying the seat situation in the current scene, and the seat information received from the seat recognition module is transmitted to the user in real time through the visual floor and area display.The application can solve the problem of seat circulation difference caused by the seat occupation phenomenon in the library.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, in particular to a seat detection and recognition system based on deep learning. BACKGROUND

[0002] Image recognition technology is a recognition technology based on the main features of an image. Each image has its features. Studies on eye movement in image recognition show that the line of sight is always focused on the main features of an image, that is, on the places where the maximum curvature or the sudden change of the outline direction of the image occurs, and the information amount of these places is the largest. Moreover, the scanning route of the eye always moves from one feature to another. It can be seen that in the image recognition process, the perceptual mechanism must exclude the redundant information of the input and extract the key information. At the same time, there must be a mechanism in the brain responsible for integrating information, which can integrate the information obtained in stages into a complete perceptual image.

[0003] In the human image recognition system, the recognition of a complex image often needs to be realized through information processing at different levels. For familiar graphics, since the main features of the graphics are mastered, the graphics are recognized as a unit without paying attention to the details. Such a whole unit composed of isolated unit materials is called a chunk, and each chunk is perceived at the same time. In the recognition of text materials, people can not only form a chunk by combining strokes or radicals of a Chinese character, but also recognize a chunk unit formed by words or phrases that often appear together.

[0004] The development of image recognition has experienced three stages: character recognition, digital image processing and recognition, and object recognition. The research on character recognition began in 1950, which generally recognizes letters, numbers and symbols, and is applied very widely from printed character recognition to handwritten character recognition.

[0005] The research on digital image processing and recognition has a history of nearly 50 years. Compared with analog images, digital images have the advantages of convenient storage and transmission, compression, non-distortion in the transmission process, and convenient processing, which provide strong power for the development of image recognition technology.

[0006] Object recognition mainly refers to the perception and understanding of objects and environment in the three-dimensional world, which belongs to the advanced computer vision category. It is a research direction combining artificial intelligence, system science and other disciplines based on digital image processing and recognition, and its research results are widely applied in various industrial and detection robots. One of the shortcomings of modern image recognition technology is poor self-adaptation. Once the target image is polluted by strong noise or the target image has a large defect, the ideal result cannot be obtained.

[0007] The mathematical essence of image recognition problem belongs to the mapping problem from pattern space to category space. At present, in the development of image recognition, there are mainly three recognition methods: statistical pattern recognition, structural pattern recognition and fuzzy pattern recognition. Image segmentation is a key technology in image processing. Since the 1970s, its research has a history of several decades, and has been highly valued by people. So far, with the help of various theories, thousands of segmentation algorithms have been proposed, and the research in this field is still actively carried out.

[0008] Due to the development of the times and the gradual improvement of people's requirements for themselves, combined with increasingly fierce competition and increasingly serious involution, leading to students' enthusiasm for focused learning gradually increasing, the call for providing a focused learning environment is getting higher and higher. Combined with the increasing enthusiasm of students for taking the public service examination and the postgraduate examination in the process of pursuing their own improvement and development, leading to a sharp increase in the number of taking the public service examination and the postgraduate examination, which also increases the demand for self-study. The increase in quantity and the fierce competition lead to the fact that the seats in the library are often in short supply, and many students cannot stay in the library all day for various reasons, and in order to prevent their own learning seats from being occupied by others, leading to the frequent occurrence of occupation behavior, and the large-scale library seat occupation phenomenon is serious.

[0009] In the case of increasing demand for self-study and high enthusiasm for self-study, the self-study industry is emerging, and some APPs help students limit the frequency of using mobile phones to improve self-study efficiency, and some businesses operate paid self-study rooms to provide good environments, etc. These all indicate that the school library cannot meet the self-study needs of many students, and the existing library self-study management system is not perfect and sound, so it is imperative to develop a library seat monitoring system to improve seat utilization and promote seat circulation.

[0010] The serious library seat occupation phenomenon leads to low seat utilization rate, and many people who really want to study have to look for a place for a long time because others occupy the seats, which wastes time; some students who are not self-studying occupy the seats for a long time, which is not conducive to seat circulation and greatly reduces the self-study benefits brought to students by each seat.

[0011] The library scene is large, with many floors and complex structure, and it is difficult for administrators to monitor such a large range. The school library management system does not have clear restrictions on occupation behavior, and it is difficult to really solve the occupation problem and warn the occupying students with only soft measures such as slogans. New students have difficulty finding a place, and there are many empty seats, but they cannot self-study because of occupation, which increases the time cost and makes the self-study students feel bad.

[0012] The data show that only a small part of students often use the seat management system, more than 90% of students occasionally use or do not use, the system usage rate is low; most students are satisfied with the seat management system, and believe that the system is helpful for the seat management of the school library. And the seat system in colleges and universities is relatively complete in function, but it can still be further improved. Nearly 30% of students think that there is no need to use the seat management system; for students who have needs, if the seat management system is implemented, they are willing to use it, and also hope that the system is more complete in function. SUMMARY

[0013] The purpose of the present application is to overcome the shortcomings and shortcomings of the prior art, and to provide a seat detection and recognition system based on deep learning, which solves the problem of seat circulation difference caused by the current library seat occupation phenomenon.

[0014] To achieve the above purpose, the technical scheme provided by the present application is: a seat detection and recognition system based on deep learning, comprising:

[0015] An image acquisition and transmission module is used to transmit the image information collected by the camera to the server, and after the server processes the real-time video image of the seat collected by the camera, part of the image frame is intercepted and processed by anti-distortion to be transmitted to the seat detection module as image features;

[0016] A seat detection module uses a pre-trained deep learning detection model to detect and segment each seat from the image features; wherein the deep learning detection model integrates Cascade-RCNN, FPN and ResNeSt network;

[0017] A seat recognition module uses a pre-trained ResNet101 network to recognize the current state of each segmented seat in real time;

[0018] A user interface module is used to visually display the seat situation in the current scene, and the seat information received from the seat recognition module is transmitted to the user in real time through the visual floor and area display.

[0019] Further, the image acquisition and transmission module collects image data through an OV2640 wide-angle lens at a frame rate of 60 and transmits it to the WIFI module through a serial port, the WIFI module accesses the ONENET protocol to complete network configuration, sets the protocol mode to realize the protection work of video data, and as a TCP client, sends video stream to the server process TCP server during local area network transmission, and finally corrects the distortion by using matrix transformation method according to the distortion parameters measured in advance by the wide-angle lens on the server side.

[0020] Further, the seat detection module adopts a deep learning detection model integrating Cascade-RCNN, FPN and ResNeSt network, wherein the Cascade-RCNN network uses ResNeSt as a multi-channel attention skeleton network to process the input image, the ResNeSt network is a latest improved model of ResNet based on attention, the ResNeSt network is improved on the basis of the inter-channel attention of the SK-Net, a plurality of Cardinal groups are added, that is, base groups, after the intra-group channel attention fusion, the features of different base groups are spliced, the feature richness of the lower layer is greatly increased, and after the Cascade-RCNN network outputs the features, a plurality of detection boxes with different IoU values are used for cascade to gradually optimize the boxes with low IoU values, and the RPN detection box module of each detector uses the FPN feature pyramid structure to perform multi-scale processing on the input features and enhance the detection ability of different scales.

[0021] Further, the seat recognition module adopts a ResNet101 network as a backbone network for seat state classification, the ResNet101 network is trained in a "pre-training-fine-tuning" framework and is initialized by using the pre-training weight based on the ImageNet data set published by the Microsoft company, prior knowledge is fully utilized, the amount of seat state data set collection is reduced, and the work difficulty is reduced, finally, a simple judgment logic is designed, that is, according to the judgment of the current seat state by the ResNet101 network and in combination with the state of the seat at a plurality of time points in the past, the final state of the current seat is determined.

[0022] Further, the user interface module is a management end interface including floor and area display, seat screening and seat state monitoring function, the user interface module can also display more detailed information of the seat state, including disabled, occupied, someone, no one and the actual state of the seat displayed as "unavailable" on the user end is "normal use" or "occupied", the scroll-view is used to realize the sliding of the floor unit and the dragging of the seat map, the function of floor and area display is completed, the seat information is stored in the MySQL database based on the Django framework, the function of seat state monitoring is completed, and the seat screening function is realized according to the monitoring result.

[0023] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0024] 1. Detection and recognition are separated, and high-frequency real-time seat state detection is realized by multiplexing a detection result and performing multiple recognitions.

[0025] 2. The system has high accuracy, high real-time performance, full-automatic detection, automatic positioning, numbering and recognition of seats, and does not need manual operation.

[0026] 3. Using unsupervised learning, it can adapt to different sites and environments with a small amount of data, and achieve simple and efficient deployment. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 is a system architecture diagram.

[0028] Figure 2 is a flowchart of seat state judgment.

[0029] Figure 3 is a detection running flowchart.

[0030] Figure 4 is a flowchart of the small program reading data. DETAILED DESCRIPTION

[0031] The application will be described in further detail below with reference to the embodiments and drawings, but the embodiments of the application are not limited thereto.

[0032] The embodiment provides a seat detection and recognition system based on deep learning, which specifically combines computer vision and deep learning technology, captures seat state information through a camera, and then judges the seat state by using image segmentation and recognition algorithm. After obtaining the specific state of the seat, the state is displayed to the user through a small program. The specific process is shown in Figure 1 , first, multiple cameras are used to collect real-time image information of seats in the library. The seats in the collected images are detected and segmented once a day to locate the specific position coordinates of the seats in the field of view of each camera on the day. After positioning, the position of the seats in the image field of view and the number on the day are sent to the subsequent small program end and classification end. After the library opens, the camera will collect images in real time, and through point-to-point transmission of local area network IP, the current frame is transmitted to the server for recognition. The server judges the current state (someone, no one, seat occupation, disable) of each corresponding numbered seat through the classification algorithm, and makes a final judgment on the seat state through the confirmation logic, and then sends the logic result to the small program end. The small program end displays the current logic state of the corresponding seat number on the small program interface, and visualizes the library state to the user.

[0033] Specifically, the system includes the following functional modules:

[0034] The seat detection module, in the data set processing, labels the collected multi-angle seat pictures, a total of 26 different angles, and performs 468 seat segmentation. 22 of them are used as the training set, and 4 are used as the verification set.

[0035] A pre-trained deep learning detection model is used to detect image features and segment each seat. This deep learning detection model integrates Cascade-RCNN, FPN, and ResNeSt networks. Cascade-RCNN uses ResNeSt as the backbone network for multi-channel attention to process the input image. ResNeSt is a recent improvement on ResNet based on attention, improving upon the inter-channel attention of SK-Net by adding multiple cardinal groups. After channel attention fusion within these groups, features from different global cardinal groups are concatenated, significantly increasing the feature richness of the lower layers. Using ResNeSt as the backbone network greatly improves object detection and segmentation performance. Cascade-RCNN is an improvement on Faster-RCNN. Compared to Faster-RCNN, which uses only a single IoU value as the criterion for bounding boxes, Cascade-RCNN outputs features and then concatenates multiple detectors with different IoU values, progressively optimizing low-IoU boxes. This effectively prevents overfitting and avoids interference between the trainer and detector. To address the IoU mismatch issue, each detector's RPN detection box module uses the FPN feature pyramid structure to process input features at multiple scales, enhancing detection capabilities at different scales. The FPN feature pyramid structure significantly improves the accuracy and sensitivity of multi-scale targets during the detection process.

[0036] The seat recognition module's logic flow is as follows: Figure 2 As shown, the actual judgment logic simplifies certain states, resulting in the final flowchart. Image segments are created using seat coordinates, and duplicate images are removed to prevent overly similar images from appearing in the training and test sets, thus avoiding data pollution. The data is labeled, resulting in 834 labeled images categorized into three states: "occupied," "empty," and "seat reserved." The dataset is divided into training and validation sets in a 4:1 ratio. The labeled data is then fed into a ResNet101 network for training. The specific process is as follows... Figure 3 As shown.

[0037] The network obtained from the data collected this time was used to verify the data collected using a mobile phone last time, and the accuracy rate was 100%. Data distribution drift did not affect the network performance.

[0038] The seat recognition module adopts a ResNet101 network as a backbone network for seat state classification, the ResNet101 network is trained in a "pre-training-fine-tuning" framework, and is initialized by using pre-training weights based on an ImageNet data set published by a Microsoft company, prior knowledge is fully utilized, the amount of collection of seat state data sets is reduced, the work difficulty is reduced, finally, a simple judgment logic is designed, that is, according to the judgment of the current seat state by the ResNet101 network and in combination with the state of the seat at a plurality of past time points, the final state of the current seat is determined;

[0039] The image acquisition and transmission module is used for transmitting the image information collected by the camera to the server. After the server processes the real-time video image of the seat collected by the camera, part of the image frames are intercepted and transmitted to the seat detection module as image features after anti-distortion processing. The hardware part places the processing chip circuit on the back, adds a heat sink to avoid overheating of the chip, fixes the power supply on the upper part of the module, and realizes the transmission of camera data to the pcb board through the data line. A certain length of data line is conducive to the flexible deployment of the camera. The antenna is also connected to the pcb board in a detachable manner to realize data transmission in poor network conditions. The direction and position of the camera can be changed at will and then fixed to facilitate the testing work. OV2640 collects image data at a rate of 60 frames and transmits it to the WIFI module through the serial port. A larger field of view and reliable clarity are the guarantee of subsequent processing work. Flexible deployment makes subsequent data collection work more convenient.

[0040] The wireless camera based on the WIFI module is adopted, and the transmission rate of the WIFI module determines the video clarity and video frame rate that can be finally obtained. A WIFI module with good performance is selected to effectively improve the overall performance. When working in a wide area network, the WIFI module is connected to the ONENET protocol to complete network configuration, and the protocol mode is set to realize the protection of video data. When working in a local area network, the WIFI module sends video stream to the server process TCP server as a TCP client.

[0041] The 130-degree wide-angle camera has a large degree of distortion, which will have a very adverse effect on subsequent processing. A point in space is finally converted from the world coordinate system to the image plane (image pixel coordinate system) during the imaging process. In this process, since the image plane is a standard rectangle, and the world coordinate system is a three-dimensional semispherical shape, the conversion process produces distortion. Through the test of the camera's own distortion parameters, the distortion can be effectively eliminated through matrix transformation

[0042] The measurement of distortion parameters is through standard calibration pictures in different positions, different angles, different postures. Then manually calibrate the chessboard node coordinate position to eliminate distortion. Use AutoCAD to design the camera shell, and design the 3D printing scheme by yourself to print.

[0043] The user interface module (i.e. the aforementioned applet side) includes the floor and area display, seat selection, seat state monitoring function management interface, which can also display more detailed information of seat state, including disabled, occupied, someone, no one and the actual state of the seat displayed as "unavailable" on the user side is "normal use" or "occupied". The applet part uses scroll-view to realize the sliding of floor units and the dragging of seat map. In the UI view writing, the design draft pixel size does not match the actual simulation model. Through unit conversion method, each module element is converted from pt to rpx, and finally the UI interface with better visual perception is obtained through fine adjustment of specific problems.

[0044] The applet contains the following functions: seat selection, floor and area display, seat state display and simple management end prototype. The seat selection function first classifies the seats and stores them as different classes, then adds bindtap function to the seat category button in WXML interface, and finally achieves the function of seat selection by adding selection logic. In terms of floor and area display, scroll-view container is used to make floor selection slider, and bindtap function and navigator navigation are added to the corresponding area and floor to achieve the purpose of jumping to the page after clicking. The seat state display is read into the applet through data request every certain period of time, and the current seat state is displayed and updated in real time through the life cycle function when jumping to this interface. The management end removes the seat selection function on the basis of the user side, adds more state information including disabled, occupied, someone, no one, etc., which is convenient for administrators to monitor the seat state in real time.

[0045] Overall, according to the design template and materials, a demo of an applet side is realized, which has 12 available seats in one floor and one area.

[0046] The MySQL database based on the Django framework is built to store seat information. The MySQL service is located in the host IP of localhost and is stored in the local. The database is provided with an app of sit, and the table for storing seat information is stored under the sit. When reading data, the Python panda and the timer timer are used to read the output seat file result.csv at regular intervals, and the seat file result.csv is stored in the table. The applet accesses the database through the interface, and the current data is written into the nowsit table every 5S. The applet front end reads the seat state variable condition from the nowsit table through the interface. The result is displayed in the applet under the support of the onshow function and the logical judgment function in the front end. The specific process is as shown in Figure 4

[0047] The above embodiments are the preferred embodiments of the present application, but the embodiments of the present application are not limited to the above embodiments, and any changes, modifications, substitutions, combinations, simplifications made without departing from the spirit and principles of the present application should be equivalent replacement methods, which are all included in the protection scope of the present application.​

Claims

1. A deep learning based seat detection and recognition system, characterized in that, The application relates to a seat detection method and device. The image acquisition and transmission module is used for transmitting image information collected by a camera to a server; after the server acquires real-time video images of seats collected by the camera, part of the image frames are intercepted, and after anti-distortion processing, the part of the image frames are transmitted to the seat detection module as image features; The seat detection module adopts a pre-trained deep learning detection model to detect and segment each seat; wherein the deep learning detection model integrates a Cascade-RCNN, a FPN and a ResNeSt network; The seat recognition module adopts a pre-trained ResNet101 network to recognize the current state of each segmented seat in real time; The user interface module is used for visually displaying the seat condition in the current scene, and the seat information transmitted by the seat recognition module is transmitted to the user in real time through visual floor and area display; The seat detection module adopts a deep learning detection model integrating a Cascade-RCNN, a FPN and a ResNeSt network; wherein the Cascade-RCNN network uses ResNeSt as a multi-channel attention skeleton network to process input images; the ResNeSt network is a latest improved model of ResNet based on attention; the ResNeSt network is improved on the basis of the inter-channel attention of the SK-Net, and multiple Cardinal groups are added, that is, groups of channel attention are fused, and then the features of different Cardinal groups are spliced, so that the feature richness of the lower layer is greatly increased; after the Cascade-RCNN network outputs the features, multiple detectors with different IoU values are used for cascade, and the low-IoU-value frame is gradually optimized; the RPN detection frame module of each detector uses the FPN feature pyramid structure to process the input features in multiple scales, and enhances the detection ability of different scales; The seat recognition module adopts a ResNet101 network as a backbone network for seat state classification; the ResNet101 network is trained in a pre-training-fine-tuning framework, and is initialized by using the pre-training weight based on the ImageNet data set published by the Microsoft Corporation; prior knowledge is fully utilized to reduce the collection amount of the seat state data set and the work difficulty; finally, simple judgment logic is designed, that is, the final state of the current seat is determined according to the judgment of the ResNet101 network on the current seat state and the state of the seat at past time. 2.The deep learning-based seat detection and identification system of claim 1, wherein: The image acquisition and transmission module collects image data through an OV2640 wide-angle lens at a frame rate of 60 frames per second and transmits the image data to a WIFI module through a serial port; the WIFI module accesses an ONENET protocol to complete network configuration, sets a protocol mode to realize protection work on video data, and when working in a local area network, sends a video stream to a TCP server as a TCP client; finally, according to the distortion parameters measured in advance by the wide-angle lens, the distortion is corrected by using a matrix transformation method. 3.The deep learning-based seat detection and identification system of claim 1, wherein: The user interface module is a management terminal interface including floor and area display, seat screening, seat state monitoring function, which can also display more detailed information of seat state, including disabled, occupied, occupied by someone, unoccupied and actual state of seat displayed as "unavailable" on the user terminal is "in normal use" or "occupied", which uses scroll-view to realize sliding of floor unit and dragging of seat map, completes the function of floor and area display, seat information is stored through MySQL database based on Django framework, completes the function of seat state monitoring, and realizes the function of seat screening according to the monitoring result.

Images