Multi-type robot and elevator dispatching method for smart park

Abstract

The invention provides a multi-type robot and elevator dispatching method for a smart park, and belongs to the technical field of neural networks. Various data acquisition devices are installed inside and outside the elevator, and the wireless communication module transmits data to the central processing system; performing denoising processing on the image data, correcting blurring by using a jitter compensation operator, performing elevator infrared filtering and updating on infrared data, and filtering an electromagnetic signal by using a multi-robot filtering transfer function; the MFDNN architecture processes the image data and the infrared data, fuses electromagnetic radiation, acceleration, pressure and image features, extracts a crowd density map and the number of people, identifies crowd key point positions and classifies the crowd key point positions; a dynamic priority scheduling algorithm optimizes robot tasks and path planning, and dynamically adjusts an elevator access control strategy and scheduling delay. According to the method, the conflict generated when the intelligent park robot and the crowd jointly use the elevator is remarkably reduced, and intelligent recognition and priority scheduling optimization of the robot and the crowd in the elevator are achieved at the same time.

Description

Technical Field

[0001] The present invention relates to the field of neural network technology, and in particular to a multi-type robot and elevator dispatching method for a smart park. Background Art

[0002] In modern smart parks, elevators, as the main means of vertical transportation, are frequently used and carry a large number of personnel and material transportation tasks. With the development of robotics technology, more and more robots are deployed in buildings to perform various tasks such as delivery, security, and cleaning. However, when these robots use elevators, they will inevitably conflict with human users, especially for some priority personnel such as infants, the elderly, and pregnant women. These conflicts may cause inconvenience and safety hazards. Specifically, priority personnel such as infants, the elderly, and pregnant women need more space and time when using elevators, and their ability to respond in emergencies is weaker. These people may face the following problems when sharing elevators with robots and other ordinary passengers: elevator congestion. During peak hours, elevators receive call requests from robots and humans at the same time. In particular, priority personnel need more time to enter and exit the elevator, which increases the elevator load and prolongs the waiting time for passengers. Safety hazards. When robots move in the elevator, they may collide with strollers, the elderly and pregnant women, increasing safety risks. User experience is degraded. Priority personnel frequently encounter robots when using the elevator, which may cause inconvenience and anxiety, reducing their satisfaction and sense of security. Task delays. The robot's task scheduling conflicts with the needs of human passengers, which may lead to prolonged task execution time or even task failure of the robot.

[0003] However, the existing scheduling between robots and elevators is simple, and does not consider the safety issues between priority personnel and robots or the efficiency of elevator use. For example, simple time segmentation: the hotel sets a specific time period for robots to use the elevator, during non-peak hours with less passenger flow, to avoid direct conflicts with human passengers. Although this method reduces conflicts to a certain extent, the demand for robots during peak hours is still difficult to meet. A simple scheduling rule: The robot scheduling system uses pre-set simple rules, such as giving priority to robots when the elevator is idle. However, this method relies on the idle state of the elevator. When the elevator is occupied for a long time during peak hours, the robot's task will still be delayed. Or there is a more direct manual scheduling method that relies on manual labor and wastes additional labor costs.

[0004] Therefore, a multi-type robot and elevator scheduling method for a smart park is needed. Summary of the invention

[0005] In view of this, the present invention provides a method for dispatching multiple types of robots and elevators in a smart park to solve the problem of elevator conflicts between multiple types of robots and people who need priority. Through intelligent scheduling and optimization algorithms, the efficiency of elevator use is improved, the conflicts between robots and humans (especially priority people) are reduced, and the overall user experience is improved. Factors such as the task priority of the robot and the urgency of human passengers are considered to ensure the efficiency, reliability and fairness of the system.

[0006] To this end, the present invention provides the following technical solutions:

[0007] A method for dispatching multiple types of robots and elevators in a smart park, comprising:

[0008] The central processing system receives data collected by each sensor through the wireless communication module and performs data preprocessing;

[0009] The central processing unit dispatches robots and elevators through the MFDNN three-stage multimodal deep neural network architecture;

[0010] MFDNN three-stage multimodal deep neural network architecture, including: robot detection network, crowd counting judgment network, robot and crowd interaction network;

[0011] The robot detection network detects electromagnetic radiation data, acceleration data A(t), pressure P o (t), image fusion feature f fused Fusion is performed to obtain the full-state fusion feature F fused ; Full-state fusion feature F fused The probability of the robot's existence is judged through the Softmax layer and the fully connected layer. When the probability exceeds the threshold, it is determined that the robot exists, otherwise it does not exist;

[0012] The crowd counting judgment network extracts the characteristics of people in the elevator, obtains the probability of each priority person through the priority person classifier, and calculates the number of priority people through the hard classification method;

[0013] The robot and crowd interaction network performs robot and elevator scheduling through information provided by the robot detection network and the crowd counting judgment network, including:

[0014] Determine the elevator's stop floors and order based on the priority of people and robots in the elevator;

[0015] There are priority people in the elevator, and the robots outside the elevator are waiting in line and delayed in dispatch;

[0016] If there are more than two priority personnel in the elevator, the robot in the elevator will stop executing its mission and limit its range of activities;

[0017] When there are no priority personnel in the elevator and the crowd density is less than the sparse crowd density value, the robot is scheduled according to the task priority.

[0018] Furthermore, the data preprocessing includes:

[0019] The data collected by the infrared sensor is filtered using the elevator infrared filter formula, and the data collected by the infrared sensor is updated using the elevator infrared filter update formula:

[0020]

[0021] in, represents the predicted state, F is the state transfer matrix, B is the control input matrix, u k-1 is the control quantity; K k is the Kalman gain, z k is the measurement value, and H is the measurement matrix.

[0022] Furthermore, the data preprocessing further includes:

[0023] The blur caused by camera image jitter is corrected by the jitter compensation operator:

[0024] v x (t) = ∫a x (t)dt

[0025] v y (t) = ∫a y (t)dt

[0026] v z (t) = ∫a z (t)dt

[0027] x(t)=∫v x (t)dt

[0028] y(t)=∫v y (t)dt

[0029] z(t)=∫v z (t)dt

[0030] Among them, a x 、a y 、a z It represents the acceleration data of the elevator in three axes obtained by the acceleration sensor installed inside the elevator, v x (t), v y (t), v z (t) represents the speed of the elevator in each direction obtained by integrating the acceleration data, and x(t), t(t), and z(t) represent the displacement of the elevator in each direction;

[0031] Update the image pixels for displacement compensation:

[0032]

[0033] Among them, x orig ,y orig is the pixel position of the original image, x new ,y new is the pixel position of the updated image, and α is an adjustment coefficient used to control the degree of compensation.

[0034] Furthermore, the data preprocessing further includes:

[0035] Convert electromagnetic field values:

[0036] F{E(t)}=E(f)

[0037] E(f) is the intensity of electromagnetic radiation at frequency f;

[0038] Electromagnetic signal filtering via multi-robot filter transfer function:

[0039]

[0040] Among them, H(t) is the parameter after multi-robot filtering, f o is the center frequency, BW is the bandwidth, and j is the imaginary unit;

[0041] Eigenvalues ​​to quantify the electromagnetic radiation of the robot:

[0042]

[0043] Where w is the weight coefficient of the nth harmonic, N is the maximum number of harmonics considered, and IEI is the robot characteristic index.

[0044] Furthermore, the robot detection network detects electromagnetic radiation data, acceleration data A(t), pressure P o (t), image fusion feature f fused Fusion is performed to obtain the full-state fusion feature F fused ,include:

[0045] Perform convolution calculation on the image data obtained by the high-precision camera inside the elevator:

[0046] C(F i )=W*F i +b

[0047] Among them, C(F i ) is the output after the first layer of convolution, W is the convolution kernel, * is the convolution operation, and b is the bias;

[0048] Elevator infrared data extraction is performed through the elevator infrared multi-scale feature fusion operator:

[0049]

[0050] Among them, IR i is the input infrared image data, representing the temperature distribution inside the elevator, W k is the convolution kernel weight matrix, b k is the bias vector, α k is the fusion weight of features of different scales, M(IR i ) is the infrared high-order feature representation after the first layer feature fusion;

[0051] Through high-order image fusion, the image data after convolution calculation is fused with the elevator infrared data after data extraction:

[0052] f fused =concat(C(F i ),M(IR i ))

[0053] Among them, concat represents the tensor concatenation operation;

[0054] For electromagnetic radiation data, acceleration data A(t), pressure P o (t), image fusion feature f fused Perform full-state fusion:

[0055] F fused =αff fused +βA(t)+γP o (t)+εIEI

[0056] Among them, α, β, γ and ε are fusion weights respectively.

[0057] Furthermore, the extraction of the features of people in the elevator and obtaining the probability of each priority person through a priority person classifier include:

[0058] Based on the elevator image, extract the elevator crowd density map:

[0059] D(x,y)=D(C(F i ))

[0060] Among them, D(x,y) is the density map, D is the density estimate, C(F i ) is the first convolution output of the elevator image;

[0061] Based on the elevator crowd density map, the total number of people N in the elevator is extracted by integral calculation:

[0062] N=∫∫D(x,y)dxdy

[0063] Based on the density map of the crowd in the elevator and the total number of people in the elevator, the key point positions of each crowd in the elevator are extracted through the human skeleton framework model:

[0064]

[0065] in is the coordinate of the jth key point of the i-th person, and M is the number of key points of each person;

[0066] The pregnant woman classifier is: P(pregnant woman) = C pre (K)

[0067] Among them, K is a single individual in the density map, C pre is the classification of pregnant women, P(pregnant women) is the probability that a single individual in the density map is a pregnant woman;

[0068] The child classifier is: P(child) = C chi (K)

[0069] Among them, K is a single individual in the density map, C pre is the child classification body, P(pregnant woman) is the probability that a single individual in the density map is a child;

[0070] The old person classifier is: P(old person) = C old (K)

[0071] Among them, K is a single individual in the density map, C pre is the classification of the elderly, and P(pregnant woman) is the probability that a single individual in the density map is an elderly person.

[0072] Furthermore, the method of determining the stopping floors and order of the elevator according to the priority of the people and robots in the elevator includes:

[0073] The priority function is used to determine the priority of different passengers and robots:

[0074] P(x)=w h H(x)+w c C(x)+w e E(x)

[0075] Among them, H(x) is the priority personnel: pregnant women, the elderly, and children, C(x) is the robot's task priority, E(x) is the emergency priority, and w h , w c , w e is the weight coefficient;

[0076] The elevator stop probability determines the elevator stop floor and order:

[0077] Stop(f)=argmaxP(x,f)

[0078] Among them, f is the target floor of the elevator. The system prioritizes high-priority passengers and robot tasks to avoid unnecessary stops.

[0079] Furthermore, there is a priority person in the elevator, and the robot outside the elevator is waiting in line and delayed in dispatch, including:

[0080] Confirm external robot priority:

[0081] Q(t)={R1(t),R2(t),R3(t),...,R n (t)}

[0082] Among them, Q(t) is the permutation result, R n (t) is the nth robot waiting outside the elevator;

[0083] Through the dynamic access control operator, the elevator access control strategy is dynamically adjusted:

[0084]

[0085] Among them, P p is the passenger priority, P r Priority for external robots;

[0086] Delayed robot dispatch:

[0087] T d =f(N,P p ,P r )

[0088] Among them, T d The delay time for robot scheduling is calculated based on the number of people and their priority.

[0089] Furthermore, if there are more than two priority personnel in the elevator, the robot in the elevator will stop executing its mission and limit its range of activities, including:

[0090]

[0091] Among them, R(t) represents the activity status of the robot, Limited means limiting the robot's activity, and Allowed means allowing the robot's activity.

[0092] Furthermore, when there is no priority person in the elevator and the crowd density is less than the sparse crowd density value, scheduling is performed according to the robot task priority, including:

[0093] Calculate the elevator crowd density:

[0094]

[0095] Where A is the area of ​​the elevator and ρ is the density distribution;

[0096] Dynamically adjust the robot's task priority and path planning:

[0097]

[0098] Among them, C i (t) is the priority of the ith robot, T j (t) is the task importance weight, and M is the number of tasks.

[0099] Advantages and positive effects of the present invention:

[0100] The present invention realizes comprehensive monitoring of the internal and external conditions of the elevator by configuring high-precision cameras, infrared sensors, weight sensors, acceleration sensors, pressure sensors and electromagnetic radiation sensors inside and outside the elevator. The data collected by the sensor is transmitted to the central processing system through the wireless communication module, and the image data is cleaned and denoised at multiple levels to ensure the accuracy of the data. The technology of removing noise through adaptive median filter, filtering infrared data through infrared filtering formula, and correcting blur caused by camera jitter through jitter compensation operator ensures the accuracy of data processing.

[0101] The MFDNN three-stage multimodal deep neural network architecture of the present invention can accurately determine whether there is a robot in the elevator and identify the characteristics of specific groups of people (such as pregnant women, children, and the elderly). Through the constructed dynamic priority scheduling algorithm, the system can dynamically adjust the processing flow according to the real-time detection results to ensure the safety of the crowd and optimize the efficiency of the elevator. In terms of elevator conflict optimization management, through technical means such as priority functions, docking probability formulas, and robot scheduling delay formulas, reasonable scheduling and management of robots and people are achieved to avoid conflicts and improve the reliability and stability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0102] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0103] Figure 1 This is a flow chart of an embodiment of the present invention;

[0104] Figure 2 This is a schematic diagram of an algorithm according to an embodiment of the present invention;

[0105] Figure 3 This is an infrared distribution diagram of an elevator according to an embodiment of the present invention;

[0106] Figure 4 Figure 2 is a diagram of a scheduling method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0107] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0108] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0109] The present invention provides a method for dispatching multiple types of robots and elevators in a smart park, which significantly reduces the conflicts between robots and people in the smart park when they use elevators together, and simultaneously realizes intelligent identification and priority dispatching optimization of robots and people in the elevators, meeting the needs for efficient, accurate and safe management in the smart park.

[0110] Arrange sensors inside and outside the elevator, including: high-precision cameras, infrared sensors, weight sensors, acceleration sensors, pressure sensors, and electromagnetic radiation sensors;

[0111] Combination Figure 1 , 2 To further illustrate, the method of the present invention comprises the following steps:

[0112] S1, transmit the data collected by the sensor to the central processing system through the wireless communication module;

[0113] S2. Perform multi-level cleaning and denoising on the image data collected by the sensor, including: denoising using an adaptive median filter; filtering the data collected by the infrared sensor using the elevator infrared filter formula; updating the data collected by the infrared sensor using the elevator infrared filter update formula, correcting the blur caused by the camera image jitter using the jitter compensation operator, and updating the image pixels using the displacement compensation update formula; converting the electromagnetic field value according to the electromagnetic conversion formula, filtering the electromagnetic signal using the multi-robot filter transfer function, and quantifying the characteristic value of the robot's electromagnetic radiation using the robot characteristic index formula;

[0114] S3. Construct a MFDNN (Multi-stage Fusion Deep Neural Network) three-stage multimodal deep neural network architecture. The MFDNN three-stage multimodal deep neural network architecture includes: robot detection network architecture, crowd counting judgment network architecture, and robot and crowd interaction network architecture. The image data obtained by the high-precision camera inside the elevator and the infrared data obtained by the infrared sensor are first convolved by the elevator fusion convolution operator, and the elevator infrared data is extracted by the elevator infrared multi-scale feature fusion operator. The image high-order fusion formula is used to fuse the image data after convolution calculation and the elevator infrared data after model extraction. A full-state fusion formula is used to combine the electromagnetic radiation data, acceleration data A(t), and pressure P o (t) and the final fusion features of the image are fully fused, and the elevator crowd density map is extracted based on the density estimation formula. The total number of people in the elevator is extracted through the integral calculation formula, and the key points of the crowd are further located through the human skeleton framework model. Finally, the number of each crowd is calculated through the pregnant woman classifier, child classifier and elderly classifier;

[0115] S4. Construct a robot and elevator scheduling strategy based on a dynamic priority scheduling algorithm. Confirm the priorities of different passengers and robots through the priority function, and determine the elevator's stop floors and order through the elevator stop probability formula. Adjust the robot's tasks or limit its activity area through the robot scheduling formula, calculate the elevator crowd density through the density formula, dynamically adjust the robot's task priority and path planning according to the robot planning formula, confirm the external robot priority through an external robot waiting queue formula, and propose a dynamic access control operator to dynamically adjust the elevator access control strategy. Finally, propose a robot scheduling delay formula to delay the robot's scheduling.

[0116] In this implementation, S1 specifically includes the following steps:

[0117] S11. Multiple high-precision cameras and infrared sensors are configured inside the elevator to cover the panoramic view inside the elevator. The image data collected by each camera is in the form of F i (t) indicates that infrared data is represented by IR i (t) represents, where i represents the camera number, t is the time, and the weight sensor W is arranged o (t);

[0118] S12. Arrange camera F in the waiting area outside the elevator o (t),real-time monitoring of the number, weight and type of pedestrians waiting for the elevator;

[0119] S13. Furthermore, an acceleration sensor A(t) is installed at the bottom of the elevator to detect the slight movement and vibration of the elevator. When an object enters the elevator, if a slight drop or displacement occurs, the acceleration sensor A(t) records the acceleration data, which can be used to analyze the movement caused by the change of the elevator load.

[0120] S14, install the pressure sensor P o (t) In the hydraulic system of an elevator, to monitor changes in system pressure, where pressure changes can reflect changes in load, especially the entry or exit of heavy objects;

[0121] S15. In the elevator, electromagnetic radiation sensors are arranged to monitor the electromagnetic field value E(t) in the surrounding environment, especially to characterize the changes caused by the motor. It is known that the unique components in the robot often emit electromagnetic radiation of a specific frequency to distinguish it from the communication equipment carried by pedestrians.

[0122] S16. Based on wireless communication, all collected data are transmitted to the central processing system of the smart park. The data is represented as: D(t) = {F i (t),W o (t),F o (t),A(t),E(t),IR i (t),P o (t)}.

[0123] In this implementation, S2 specifically includes the following steps:

[0124] S21. Perform multi-level data cleaning and denoising on the collected image data in the central processing system of the smart park. First, an adaptive median filter M(f) is used to denoise the image data to filter out environmental noise and artifacts introduced by camera shaking caused by objects entering and exiting the elevator. The window size of the median filter is automatically adjusted according to the noise level to retain image details. Then, the infrared sensor data IR i(t) performing elevator infrared filtering to remove random noise and measurement errors in temperature data;

[0125] Among them, the elevator infrared filter is expressed as follows:

[0126]

[0127] in, represents the predicted state, F is the state transfer matrix, B is the control input matrix, u k-1 To control the amount;

[0128] Furthermore, the elevator infrared filter update formula is:

[0129]

[0130] Among them, K k is the Kalman gain, z k is the measurement value, and H is the measurement matrix.

[0131] S22. In order to eliminate the artifacts introduced by the camera shaking caused by the elevator shaking due to objects entering and exiting the elevator, a jitter compensation operator is proposed to adjust the image captured by the camera according to the data of the acceleration sensor to reduce the blur and displacement caused by the jitter;

[0132] v x (t) = ∫a x (t)dt

[0133] v y (t) = ∫a y (t)dt

[0134] v z (t) = ∫a z (t)dt

[0135] x(t)=∫v x (t)dt

[0136] y(t)=∫v y (t)dt

[0137] z(t)=∫v z (t)dt

[0138] Among them, a x 、a y and a z It means that the acceleration sensor installed inside the elevator obtains the acceleration data of the elevator in three axes, v x (t), v y (t) and v z(t) represents the speed of the elevator in each direction by integrating the acceleration data, and x(t), y(t) and z(t) represent the displacement of the elevator in each direction;

[0139] According to the relative position of the camera and the jitter of the elevator, the displacement compensation of the camera is calculated, and the displacement compensation update formula is proposed to update the image pixels:

[0140]

[0141] Among them, x orig ,y orig is the pixel position of the original image, x new ,y new is the pixel position of the updated image, and α is an adjustment coefficient used to control the degree of compensation;

[0142] S23. In the central processing unit of the smart park, the electromagnetic field value E(t) is converted according to the electromagnetic conversion formula:

[0143] F{E(t)}=E(f)

[0144] Where E(f) is the intensity of electromagnetic radiation at frequency f.

[0145] The frequency distribution of electromagnetic radiation is revealed by means of the above-mentioned spectrum analysis.

[0146] In an elevator environment, there are usually a variety of electromagnetic interferences, so the collected signals need to be filtered to remove irrelevant noise. Therefore, a multi-robot filter transfer function is proposed:

[0147]

[0148] Among them, H(t) is the parameter after multi-robot filtering, f o is the center frequency, BW is the bandwidth, and j is the imaginary unit.

[0149] The characteristic value of the robot's electromagnetic radiation is quantified through the robot characteristic index formula:

[0150]

[0151] Among them, w is the weight coefficient of the nth harmonic, N is the maximum harmonic number, and IEI is the robot characteristic index;

[0152] Through the above robot characteristic index formula, the system can analyze the electromagnetic radiation characteristics in the elevator environment in real time and distinguish robots from ordinary pedestrians in a certain physical sense;

[0153] S24. Normalize all data in the above-mentioned smart park central processing system:

[0154]

[0155] Among them, X is the original data, X min and X max are the minimum and maximum values ​​of the data set, respectively.

[0156] S3. Utilize sensor data to further identify the characteristics of passengers through deep learning and crowd technology, and calculate the number of priority personnel:

[0157] S31. Build the robot detection network architecture:

[0158] S311, the input layer is constructed including the elevator internal image data, electromagnetic radiation data, acceleration data A(t) and pressure P processed in step S2 o (t);

[0159] S312, image data extraction layer, specifically including:

[0160] An elevator fusion convolution operator is proposed for the image data F inside the elevator. i (t) Perform convolution calculation:

[0161] C(F i )=W*F i +b

[0162] Where C(F i ) is the output after the first layer of convolution, W is the convolution kernel, * is the convolution operation, and b is the bias;

[0163] Convolution extraction of infrared data inside the elevator is performed. A major feature that distinguishes robots from pedestrians, pregnant women and children is that they emit less heat. Therefore, an elevator infrared multi-scale feature fusion operator is used to extract elevator infrared data:

[0164]

[0165] Among them, IR i is the input infrared image data, representing the temperature distribution inside the elevator, W k is the convolution kernel weight matrix, b k is the bias vector, α k is the fusion weight of features of different scales, M(IR i ) is the infrared high-order feature representation after the first layer feature fusion.

[0166] The image data after convolution calculation and the elevator infrared data after model extraction are fused through the high-order image fusion formula:

[0167] f fused=concat(C(F i ),M(IR i ))

[0168] F fused It is the feature representation of the final fused image, and concat represents the tensor concatenation operation.

[0169] S313, fusion layer construction, the modal data in step S311 and step S312 are fused through the full-state fusion formula, which are electromagnetic radiation data, acceleration data A(t), pressure P o (t) and the final fusion feature representation of the image:

[0170] F fused =αf fused +βA(t)+γP o (t)+εIEI

[0171] Among them, α, β, γ, and ε are fusion weights.

[0172] Step S314: construct a fully connected layer FC (Full-Connect), and judge the probability of the robot's existence through the Softmax layer and the fully connected layer. When the probability exceeds the threshold, it is determined that the robot exists;

[0173] In this way, it is possible to quickly and accurately identify whether there is a robot inside the park elevator.

[0174] S32. Construct a network layer for counting and judging people inside the elevator. This module focuses on identifying the density, distribution, and specific characteristics of the crowd (such as pregnant women, children, etc.).

[0175] Extract the elevator crowd density map D(x,u) based on the density estimation formula:

[0176] D(x,y)=D(C(F i ))

[0177] Extract the total number of people N in the elevator through the integral calculation formula:

[0178] N=∫∫D(x,y)dxdy

[0179] Extract the key point positions of each person inside the elevator based on the human skeleton model:

[0180]

[0181] in is the coordinate of the jth key point of the ith person, and M is the number of key points for each person.

[0182] Using pregnant woman classifier C pre To get a pregnancy test:

[0183] P(pregnant woman)=C pre (K)

[0184] Using the children classifier C chi Testing for children:

[0185] P(children)=C chi (K)

[0186] Using the elderly classifier C old Testing for children:

[0187] P(old man)=C old (K)

[0188] Based on the hard classification method, the number of each priority personnel is directly calculated;

[0189] In the crowd technical judgment network layer, the main task of the system is to identify the characteristics of passengers in the elevator and to judge the density and distribution of the crowd. The system can accurately identify the characteristics of passengers in the elevator, especially those groups that need priority protection, such as pregnant women, children and the elderly. These provide an accurate data basis for subsequent priority calculations and scheduling decisions. The system calculates the total number of people and the crowd density map in the elevator based on the data from the camera and infrared sensor. This information directly affects the operation strategy of the elevator and the priority setting of the robot scheduling. Based on the human skeleton framework model, the system identifies the key point position of each passenger. This helps to judge the physical state of the passenger, and can also identify the posture and behavior of the passenger in order to make better scheduling decisions. The crowd technical judgment network layer further subdivides the characteristics of the passengers through the pregnant woman, child and elderly classifiers, so as to better evaluate the priority, which is helpful for formulating reasonable scheduling strategies when there are many special groups (pregnant women or the elderly).

[0190] In this implementation, S4 specifically includes the following steps:

[0191] S41. When there are pregnant women, elderly people, children and robots in the elevator, the system needs to prioritize the safety and comfort of the people. By adjusting the operation strategy of the elevator to avoid direct contact between people and robots, the priority function P(x) is first defined to represent the priority of different passengers and robots:

[0192] P(x)=w h H(x)+w c C(x)+w e E(x)

[0193] Among them, H(x) is the priority of the crowd (pregnant women, the elderly, and children), C(x) is the task priority of the robot, E(x) is the emergency priority, and w h , wc , w e is the weight coefficient.

[0194] S42. Based on the priority function P(x), the elevator's stop floors and order are determined by the elevator stop probability formula:

[0195] Stop(f)=argmaxP(x,f)

[0196] Where f is the elevator's target floor. The system prioritizes high-priority passengers and robot tasks to avoid unnecessary stops.

[0197] S43. When it is detected that there may be a conflict between the crowd and the robot, that is, the total number of pregnant women, children, and elderly people in the crowd is greater than 2, the system adjusts the robot's task or limits its activity area through the robot scheduling formula:

[0198]

[0199] Among them, R(t) represents the activity status of the robot, Limited represents limiting the robot's activity, and Allow represents allowing the robot's activity.

[0200] S44. When there are no pregnant women, elderly people or children in the elevator, the robot and other passengers are flexibly dispatched to optimize the efficiency of the elevator;

[0201] Calculate the elevator crowd density using the density formula:

[0202]

[0203] Where A is the area of ​​the elevator and ρ is the density distribution;

[0204] When the ρ density distribution is less than the set sparse crowd density value, the robot task optimization is performed. The system dynamically adjusts the robot's task priority and path planning according to the robot planning formula, giving priority to high-level robot tasks:

[0205]

[0206] Among them, C i (t) is the priority of the ith robot, T j (t) is the task importance weight, and M is the number of tasks.

[0207] S45: When there is no robot inside the elevator, and S32 determines that there are pregnant women, elderly people, and children in the elevator, and there are robots waiting to use the elevator outside, the system should give priority to ensuring the safety of the people and arrange the use of the robots reasonably:

[0208] The system first maintains an external robot waiting queue formula, sorted by priority:

[0209] Q(t)={R1(t),R2(t),R3(t),...,R n (t)}

[0210] A dynamic access control operator is used to control the opening and closing of the elevator door to prevent people and robots from entering and exiting the elevator at the same time. The access control strategy is dynamically adjusted according to the status inside and outside the elevator:

[0211]

[0212] Among them, P p is the passenger priority, P r Priority for external robots;

[0213] A robot dispatch delay formula is used to reasonably delay the dispatch of robots and avoid direct contact while ensuring the safety of passengers:

[0214] T d =f(N,p p ,P r )

[0215] Among them, T d The delay time for robot scheduling is calculated based on the number of people and their priority.

[0216] This implementation uses a MFDNN three-stage multimodal deep neural network architecture, and the system can quickly and accurately determine whether there is a robot in the elevator to avoid false alarms or missed alarms; at the same time, the system enters the second stage of the crowd technology judgment network layer. In this stage, the lightweight network can quickly identify specific crowd characteristics through key node detection, and when there is a robot, the robot and crowd interaction network in the third stage is entered to avoid recognition errors. In this way, in the scenario of smart parks and limited computing resources, the design of the three-stage architecture enables the system to quickly and dynamically adjust the processing flow according to real-time detection results to avoid unnecessary computing overhead.

[0217] In summary, this method realizes the intelligent identification and conflict optimization management of robots and people in elevators through the MFDNN three-stage multimodal deep neural network architecture and dynamic priority scheduling algorithm. The system can accurately determine whether there is a robot in the elevator, quickly identify the characteristics of specific people, and dynamically adjust the processing flow according to the real-time detection results to ensure the safety and comfort of the people. At the same time, the system improves the efficiency of elevator use and optimizes the execution of robot tasks through a flexible scheduling mechanism. This solution shows significant innovation and practicality in the scenarios of smart parks and limited computing resources, and provides a reliable guarantee for the coordinated coexistence of smart devices and people.

[0218] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for dispatching multiple types of robots and elevators in a smart park, characterized in that: include: The central processing system receives data collected by each sensor through the wireless communication module and performs data preprocessing; The central processing unit dispatches robots and elevators through the MFDNN three-stage multimodal deep neural network architecture; MFDNN three-stage multimodal deep neural network architecture, including: robot detection network, crowd counting judgment network, robot and crowd interaction network; The robot detection network detects electromagnetic radiation data, acceleration data A(t), pressure P o (t), image fusion feature f fused Fusion is performed to obtain the full-state fusion feature F fused ; Full-state fusion feature F fused The probability of the robot's existence is judged through the Softmax layer and the fully connected layer. When the probability exceeds the threshold, it is determined that the robot exists, otherwise it does not exist; The crowd counting judgment network extracts the characteristics of people in the elevator, obtains the probability of each priority person through the priority person classifier, and calculates the number of priority people through the hard classification method; The robot and crowd interaction network performs robot and elevator scheduling through information provided by the robot detection network and the crowd counting judgment network, including: Determine the elevator's stop floors and order based on the priority of people and robots in the elevator; There are priority people in the elevator, and the robots outside the elevator are waiting in line and delayed in dispatch; If there are more than two priority personnel in the elevator, the robot in the elevator will stop executing its mission and limit its range of activities; When there are no priority personnel in the elevator and the crowd density is less than the sparse crowd density value, the robot is scheduled according to the task priority.

2. A method for dispatching multiple types of robots and elevators in a smart park according to claim 1, characterized in that: The data preprocessing includes: The data collected by the infrared sensor is filtered using the elevator infrared filter formula, and the data collected by the infrared sensor is updated using the elevator infrared filter update formula: in, represents the predicted state, F is the state transfer matrix, B is the control input matrix, u k-1 is the control quantity; K k is the Kalman gain, z k is the measurement value, and H is the measurement matrix.

3. A method for dispatching multiple types of robots and elevators in a smart park according to claim 1, characterized in that: The data preprocessing further includes: Correct the blur caused by camera image jitter through the jitter compensation operator: v x (t)=∫a x (t)dt v y (t)=∫a y (t)dt v z (t)=∫a z (t)dt x(t)=∫v x (y)dt y(t)=∫v y (t)dt z(t)=∫v z (t)dt Among them, a x 、a y 、a z It represents the acceleration data of the elevator in three axes obtained by the acceleration sensor installed inside the elevator, v x (t), v y (t), v z (t) represents the speed of the elevator in each direction obtained by integrating the acceleration data, and x(t), y(t), and z(t) represent the displacement of the elevator in each direction; Update the image pixels for displacement compensation: Among them, x orih ,y orig is the pixel position of the original image, x new ,y new is the pixel position of the updated image, and α is an adjustment coefficient used to control the degree of compensation.

4. A method for dispatching multiple types of robots and elevators in a smart park according to claim 1, characterized in that: The data preprocessing further includes: Convert electromagnetic field values: F{E(t)}=E(f) E(f) is the intensity of electromagnetic radiation at frequency f; Electromagnetic signal filtering via multi-robot filter transfer function: Among them, H(t) is the parameter after multi-robot filtering, f o is the center frequency, BW is the bandwidth, and j is the imaginary unit; Eigenvalues ​​to quantify the electromagnetic radiation of the robot: Where w is the weight coefficient of the nth harmonic, N is the maximum number of harmonics considered, and IEI is the robot characteristic index.

5. The method for dispatching multiple types of robots and elevators in a smart park according to claim 1, characterized in that: The robot detection network detects electromagnetic radiation data, acceleration data A(t), pressure P o (t), image fusion feature f fused Fusion is performed to obtain the full-state fusion feature F fused ,include: Perform convolution calculation on the image data obtained by the high-precision camera inside the elevator: C(F i )=W*F i +b Among them, C(F i ) is the output after the first layer of convolution, W is the convolution kernel, * is the convolution operation, and b is the bias; Elevator infrared data extraction is performed through the elevator infrared multi-scale feature fusion operator: Among them, IR i is the input infrared image data, representing the temperature distribution inside the elevator, W k is the convolution kernel weight matrix, b k is the bias vector, α k is the fusion weight of features of different scales, M(IR i ) is the infrared high-order feature representation after the first layer feature fusion; Through high-order image fusion, the image data after convolution calculation is fused with the elevator infrared data after data extraction: f fused =concat(C(F i ),M(IR i )) Among them, concat represents the tensor concatenation operation; For electromagnetic radiation data, acceleration data A(t), pressure P o (t), image fusion feature f fuded Perform full-state fusion: F fused =αf fused +βA(t)+γP o (t)+εIEI Among them, α, β, γ and ε are fusion weights respectively.

6. A method for dispatching multiple types of robots and elevators in a smart park according to claim 1, characterized in that: The extracting of the features of people in the elevator and obtaining the probability of each priority person through a priority person classifier include: Based on the elevator image, extract the elevator crowd density map: D(x,y)=D(V(F i )) Among them, D(x,y) is the density map, D is the density estimate, C(F i ) is the first convolution output of the elevator image; Based on the elevator crowd density map, the total number of people N in the elevator is extracted by integral calculation: N=∫∫D(x,t)dxdy Based on the density map of the crowd in the elevator and the total number of people in the elevator, the key point positions of each crowd in the elevator are extracted through the human skeleton framework model: in is the coordinate of the jth key point of the i-th person, and M is the number of key points of each person; The pregnant woman classifier is: P(pregnant woman) = C pre (K) Among them, K is a single individual in the density map, C pre is the classification of pregnant women, P(pregnant women) is the probability that a single individual in the density map is a pregnant woman; The child classifier is: P(child) = C chi (K) Among them, K is a single individual in the density map, C pre is the child classification body, P(pregnant woman) is the probability that a single individual in the density map is a child; The old person classifier is: P(old person) = C old (K) Among them, K is a single individual in the density map, C pre is the classification of the elderly, and P(pregnant woman) is the probability that a single individual in the density map is an elderly person.

7. The method for dispatching multiple types of robots and elevators in a smart park according to claim 1, characterized in that: Determining the stopping floors and order of the elevator according to the priority of the people and robots in the elevator includes: The priority function is used to determine the priority of different passengers and robots: P(x)=w h H(x)+w c C(x)+w e E(x) Among them, H(x) is the priority personnel: pregnant women, the elderly, and children, C(x) is the robot's task priority, E(x) is the emergency priority, and w h , w c , w e is the weight coefficient; The elevator stop probability determines the elevator stop floor and order: Stop(f)=argmax P(x,f) Among them, f is the target floor of the elevator. The system prioritizes high-priority passengers and robot tasks to avoid unnecessary stops.

8. The method for dispatching multiple types of robots and elevators in a smart park according to claim 1, characterized in that: There are priority personnel in the elevator, and the robots outside the elevator are waiting in line and delayed in dispatch, including: Confirm external robot priority: Q(t)={R1(t),R2(t),R3(t),...,R n (t)} Among them, Q(t) is the permutation result, R n (t) is the nth robot waiting outside the elevator; Through the dynamic access control operator, the elevator access control strategy is dynamically adjusted: Among them, P p is the passenger priority, P r Priority for external robots; Delayed robot dispatch: T d =f(N,P p ,P r ) Among them, T d The delay time for robot scheduling is calculated based on the number of people and their priority.

9. The method for dispatching multiple types of robots and elevators in a smart park according to claim 1, characterized in that: If there are more than two priority personnel in the elevator, the robot in the elevator will stop executing its mission and limit its range of activities, including: Among them, R(t) represents the activity status of the robot, Limited means limiting the robot's activity, and Allowed means allowing the robot's activity.

10. The method for dispatching multiple types of robots and elevators in a smart park according to claim 1, characterized in that: When there is no priority person in the elevator and the crowd density is less than the sparse crowd density value, the robot is scheduled according to the task priority, including: Calculate the elevator crowd density: Where A is the area of ​​the elevator and ρ is the density distribution; Dynamically adjust the robot's task priority and path planning: Among them, C i (t) is the priority of the ith robot, T j (t) is the task importance weight, and M is the number of tasks.

Images