Unmanned business hall team system based on artificial intelligence and working method

Abstract

The invention discloses an unmanned business hall team system based on artificial intelligence and a working method, and belongs to the field of artificial intelligence. The system further comprises a path planning module connected in the intelligent service robot, an intelligent scheduling module electrically connected with the path planning module, a central control analysis and display module electrically connected with the intelligent scheduling module, and an environment intelligent sensing module electrically connected with the central control analysis and display module. And the central control analysis and display module is electrically connected with the intelligent control module. According to the unmanned business hall team system based on artificial intelligence and the working method provided by the invention, scheduling of the intelligent service robot team can be realized according to actual conditions, a supervision function of the unmanned business hall is realized through the intelligent service robot team, the intelligence and automation of the robot team are improved, and the working efficiency of the robot team is improved. And efficient, convenient and intelligent service experience can be provided.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence, and particularly relates to an unmanned business hall team system and working method based on artificial intelligence. Background Art

[0002] With the development of artificial intelligence technology, many services have changed from offline services to a combination of online and offline services. For offline services, they are gradually transforming from manual services to intelligent service robot services. Taking the power supply business hall as an example, the power supply business hall refers to a service window or office set up by a power supply enterprise, and its main function is to provide a series of services for power customers, which plays an important role in ensuring the stability of power supply, improving service quality, and safeguarding customer rights and interests. The traditional service method of the power supply business hall is manual offline service. Although it has been upgraded to a combination of online and offline in the later stage, with the development of society, the business of the power supply business hall is increasing. It is basically impossible for offline manual services to perfectly handle all businesses, and most business halls are equipped with a small number of staff. When there are more businesses, it is far from being able to supply the visiting customers, resulting in problems such as customer backlog and too long waiting time.

[0003] To overcome the above problems, in terms of offline services, manual services are gradually transformed into intelligent service robot services to process the business that customers need to handle more quickly. Usually, it is achieved through an intelligent service robot team. However, the existing intelligent service robot teams in unmanned business halls can usually only handle business and are difficult to schedule and supervise when there are a large number of customers. For example, it is difficult to allocate an appropriate number of intelligent service robots to the corresponding areas to help customers handle business according to the number of customers in different areas, and it is difficult to supervise the situation in the business hall. As a result, the intelligence and automation of the intelligent service robots in the unmanned business hall are relatively low. Therefore, an intelligent service robot team system that can implement scheduling and supervision functions is needed, and the present invention solves this technical problem. Summary of the Invention

[0004] The present invention provides an unmanned business hall team system and working method based on artificial intelligence, which can realize the scheduling of the intelligent service robot team according to the actual situation, and realize the supervision function of the unmanned business hall through the intelligent service robot team, improving the intelligence and automation of the robot team, and helping to provide an efficient, convenient and intelligent service experience.

[0005] An artificial intelligence-based unmanned business hall team system includes multiple intelligent service robots, and also includes a path planning module connected to the intelligent service robots, an intelligent scheduling module electrically connected to the path planning module, a central control analysis and display module electrically connected to the intelligent scheduling module, and an environmental intelligent perception module electrically connected to the central control analysis and display module. The central control analysis and display module is electrically connected to an intelligent control module;

[0006] The intelligent scheduling module is used for scheduling the intelligent service robots. The central control analysis and display module is used for analyzing and displaying data. The environmental intelligent perception module is used for identifying and supervising the environment. The intelligent control module is used for guiding customers and controlling the start and stop of hardware devices.

[0007] The present invention also provides a working method for an artificial intelligence-based unmanned business hall team. Based on the artificial intelligence-based unmanned business hall team system as described above, it specifically includes the following steps:

[0008] S1. Establish a work uniform detection model for identifying whether staff wear work uniforms as required;

[0009] S2. Establish a personnel smoking model for detecting and identifying whether there is a smoking situation among personnel;

[0010] S3. Establish a personnel falling detection model for monitoring whether there is a situation where a person suddenly falls to the ground;

[0011] S4. Establish a pedestrian flow detection model for detecting and counting the number of people in the business hall;

[0012] S5. Monitor and judge whether there are difficulties during the process of customers handling business, and guide the customers.

[0013] S6. Transmit the data collected by the intelligent service robots to the central control analysis and display module. The central control analysis and display module analyzes the obtained data and displays it on the screen;

[0014] S7. According to the data analysis situation, schedule the intelligent service robots through the intelligent scheduling module so that the intelligent service robots move to the corresponding positions to work.

[0015] Further, the step S1 includes the following steps:

[0016] S11. Collect a large amount of image data including those wearing work uniforms and those not wearing work uniforms, annotate the images, mark the specific positions of the work uniforms, and perform standardization processing on the images. Then, train the model based on the standardized images to form a work uniform detection model;

[0017] S12. The camera of the intelligent service robot captures the surrounding environment, and the model extracts local features in the image through a multi-layer convolutional neural network to obtain a feature map. The specific formula is as follows:

[0018]

[0019] Among them, I(x + i, y + j) is the input image pixel value, K(i, j) is the convolution kernel, f(x, y) is the feature output, and i and j are spatial coordinates;

[0020] S13. Generate candidate bounding boxes in the area of the feature map that may contain tooling through RPN, and calculate the object score and coordinate offset of each candidate bounding box through a loss function. The calculation formula of the loss function is as follows:

[0021] L = L cls + λ·L reg

[0022] Among them, L cls represents the classification loss, which is used to calculate the object score, and L reg represents the regression loss, which is used to calculate the coordinate offset;

[0023] S14. The classification loss L cls evaluates whether the object score is accurate by using the cross-entropy loss. The specific formula is as follows:

[0024]

[0025] S15. The classification loss L reg uses the Smooth L1 loss to optimize the position of the bounding box, which is represented by the following formula:

[0026]

[0027] S16. Perform non-maximum suppression on the obtained results to remove overlapping bounding boxes. First, calculate the intersection over union (IoU) of two bounding boxes A and B to measure the overlapping degree of these two bounding boxes:

[0028]

[0029] Then, if the IoU of two bounding boxes is greater than the set threshold, keep the bounding box with the higher object score and remove the bounding box with the lower object score, thereby giving the specific position of the tooling in the image in the form of a bounding box.

[0030] Furthermore, the step S2 includes the following steps:

[0031] S21. Use a human pose estimation model to obtain the position coordinates (x, y) of the key points of the human body, and minimize the position error through a loss function:

[0032]

[0033] Among them, y i represents the true key point coordinates, and represents the predicted key point coordinates;

[0034] S22. For the recognition of smoking actions, 3D-CNN is used to extract features in the time and space dimensions. The 3D convolution calculation formula is as follows:

[0035]

[0036] Among them, I(x+i, y+j, z+k) represents the input frame, and K(i, j, k) represents the 3D convolution kernel;

[0037] S23. Temporal analysis is performed through a long short-term memory network, and the state is updated specifically through the following formula:

[0038] Forget gate: f t =σ(W f ·[h t-1 , x t +b f )

[0039] Among them, W f represents the weight matrix, h t-1 represents the short-term memory, x t represents the current input, b f represents the bias term, and σ represents the Sigmoid function;

[0040] Input gate: i t =σ(W i ·[h t-1 , x t +b i )

[0041]

[0042] Among them, represents the candidate cell state;

[0043] Cell state update:

[0044] Among them, ⊙ represents element-wise multiplication, and C t-1 represents the cell state at the previous moment;

[0045] Output gate: o t =σ(W o ·[h t-1 , x t +b o )

[0046] h t = o t ⊙tanh(C t )

[0047] where h t represents the feature vector for classification and is used for classifying smoking behavior.

[0048] Furthermore, step S3 includes the following steps:

[0049] S31. Detect human key points through a pose estimation model;

[0050] S32. Calculate the distance change of the key points:

[0051]

[0052] S33. Calculate the angle change of the key points:

[0053]

[0054] where A and B are vectors formed by adjacent key points respectively;

[0055] S34. When a specific angle is lower than the set threshold and the distance meets the falling-down feature, it is determined as falling down, and the determination is made through the following formula:

[0056]

[0057] where w is the weight, x i is the input feature vector, and y i is the label.

[0058] Furthermore, step S4 includes the following steps:

[0059] S41. Perform object detection: Generate a bounding box through global image partitioning and object detection, and the bounding box regression formula is as follows:

[0060] Regression of the center point coordinates:

[0061] t x = σ(p x ) + c x

[0062] t y = σ(p y ) + c y

[0063] where p x and p y are prediction parameters representing the center point offset, and c x and cy is the upper left corner coordinates of the cell;

[0064] Width and height regression:

[0065]

[0066] where p w and p h are prediction parameters, representing the width and height of the prior box, b w and b h represent the scaling factors of the width and height;

[0067] S42. Perform multi-object tracking, predict and update the state through Kalman filtering, and the predicted state equation is as follows:

[0068] x t|t-1 = F·x t-1|t-1 + B·u t-1

[0069] where x t|t-1 represents the predicted state at the current moment, F is the state transition matrix, x t-1|t-1 represents the state estimate at the previous moment, B represents the control matrix, and u t-1 represents the control input;

[0070] Update equation:

[0071] x t|t = x t|t-1 + K t (z t - H·x t|t-1 )

[0072] where x t|t represents the updated state at the current moment, K t is the Kalman gain, H is the observation matrix, and z t represents the observed value at the current moment;

[0073] S43. Use the Hungarian algorithm for matching to perform optimal matching between the detection box and the tracking box, and define the Euclidean distance matrix between the targets:

[0074]

[0075] where p i and q j represent the center coordinates of different targets, and the association between detection and tracking is achieved through the Hungarian algorithm.

[0076] Furthermore, the step S5 includes the following steps:

[0077] S51. The intelligent service robot collects the actions of customers during business handling, the voice prompts of the business terminal, and the prompts on the business terminal screen, and uses machine learning algorithms to determine whether the customers have performed the correct operations;

[0078] S52. When it is detected that the actions of the customers do not meet the requirements of the current operation steps, the intelligent service robot will trigger a feedback mechanism to prompt the customers to correct their actions;

[0079] S54. Connect the power-on / off hardware module to the business device for remotely controlling the power-on / off of the business device.

[0080] Further, step S5 further includes the following steps:

[0081] S54. Establish a troubleshooting model;

[0082] S55. Collect the images displayed on the business device interface and judge the problems existing in the business device;

[0083] S56. Provide feedback to the customers according to the judgment results.

[0084] Further, step S6 includes the following steps:

[0085] S61. The data collected by the intelligent service robot is uploaded to the central control analysis and display module;

[0086] S62. The collected data is displayed on the screen in real time;

[0087] S63. Automatically generate reports based on the collected data.

[0088] Further, step S7 includes the following steps:

[0089] S71. According to the data of the surrounding environment collected by the intelligent service robot, the intelligent scheduling module schedules the idle intelligent service robots to the designated positions;

[0090] S72. The route planned by the intelligent scheduling module is transmitted to the intelligent service robot, and the intelligent service robot can avoid obstacles according to the situation of nearby obstacles.

[0091] The technical effects of the present invention are as follows:

[0092] (1) The intelligent service robot in this solution can collect the surrounding environmental conditions and upload them to the environmental intelligent perception module for analyzing the collected image data. Then, the data is transmitted to the central control analysis and display module. When there is an abnormal situation, it will be displayed on the large screen, and the intelligent service robot will be scheduled through the intelligent scheduling module to move to the corresponding position for guidance or warning, thus realizing the effective supervision of the unmanned business hall and being able to serve customers in a timely manner, making the intelligent service robot team more intelligent and automated, and improving the user experience;

[0093] (2) The environmental intelligent perception module in this solution can detect whether the staff is wearing work clothes through the work clothes detection model, judge whether there is a behavior of staff smoking through the personnel smoking model, detect whether the people in the business hall suddenly fall to the ground through the personnel falling to the ground model, and discover whether there is a crowded situation through the crowd flow detection model, so as to supervise the situation in the business hall from multiple aspects, enabling the intelligent service robot to not only complete regular services but also achieve multi-functional effects, making the service more comprehensive;

[0094] (3) The information collected by the intelligent service robot in this solution can be uploaded to the central control analysis and display module, which can be used to count data such as abnormal situations that occur, business handling time, and business handling times, facilitating the subsequent optimization of business services and being conducive to providing a more efficient, convenient, and intelligent service experience in the future. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] Figure 1 is the connection block diagram of the present invention.

[0096] Figure 2 is the process schematic diagram of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0097] The technical solutions of the present invention will be clearly and completely described below in conjunction with specific embodiments and the accompanying drawings.

[0098] See Figure 1 , an unmanned business hall team system based on artificial intelligence, including multiple intelligent service robots, and further including a path planning module connected in the intelligent service robot, an intelligent scheduling module electrically connected to the path planning module, a central control analysis and display module electrically connected to the intelligent scheduling module, an environmental intelligent perception module electrically connected to the central control analysis and display module, and the central control analysis and display module is electrically connected to the intelligent control module;

[0099] The intelligent scheduling module is used for the scheduling of intelligent service robots. The central control analysis and display module is used for data analysis and display. The environmental intelligent perception module is used for environmental identification and supervision. The intelligent control module is used to guide customers and control the start and stop of hardware devices.

[0100] Preferably, in the intelligent service robot team of this embodiment, an intelligent navigator is arranged, that is, the "team leader" in the team. The intelligent navigator itself also belongs to an intelligent service robot and is equipped with an intelligent scheduling module for scheduling other intelligent service robots in the team. Taking the unmanned business hall as an example, the intelligent navigator in this embodiment is located at the front desk and is mainly responsible for greeting and reception. Other robots are arranged in different areas to be responsible for different tasks respectively, such as the business handling area, the exhibition hall introduction area, the payment area, the customer waiting area, etc. When there are many customers in a certain area and it is busy, the intelligent navigator can, according to the flow of people, dispatch the idle intelligent service robots in other areas to the busy area for auxiliary operations, so that the intelligent service robots can flexibly adapt to different task types and working scenarios, as well as rapidly changing demands, and achieve highly personalized services.

[0101] The intelligent service robot in this embodiment is equipped with components such as a camera, a lidar, and a walking mechanism for functions such as visual navigation and obstacle avoidance. Its specific structure can adopt the structure of an intelligent robot in the prior art as long as it has the above components. Since the specific structure of the intelligent service robot is not the innovation point of the present invention and its structure can adopt the structure of a robot in the prior art, its specific structure will not be described in detail here.

[0102] The scheduling principle of the intelligent scheduling module in this solution can refer to the scheduling principle in the prior art. For example, for the invention patent with the application number 202011490180.8 and the name of a robot path planning and scheduling method, the intelligent scheduling module can refer to the scheduling background recorded therein to perform path planning and scheduling on other intelligent service robots. The planned path can be transmitted to the corresponding intelligent service robot through the network. When the intelligent service robot moves along the planned route, it can also further plan the path through the path planning module according to the situation of the surrounding obstacles collected, so as to achieve autonomous obstacle avoidance. Thus, the intelligent service robot can not only travel along the scheduling path during driving, but also still maintain autonomous obstacle avoidance according to the observed situation of the nearby obstacles. Preferably, the planning principle of the path planning module in this embodiment can be implemented by the dynamic window algorithm, which is a prior art well-known to those skilled in the art and will not be described in detail here.

[0103] This system can dispatch other intelligent service robots through the intelligent navigator according to the situation of the surrounding environment collected by the environmental intelligent perception module, so as to achieve better services.

[0104] See Figure 2 The present invention also provides a working method for the unmanned business hall team based on artificial intelligence. Based on the unmanned business hall team system based on artificial intelligence as described above, the method specifically includes the following steps:

[0105] S1. Establish a work uniform detection model for identifying whether the staff wears the work uniform as required;

[0106] S2. Establish a personnel smoking model for detecting and identifying whether there is a smoking situation among personnel;

[0107] S3. Establish a personnel falling detection model for monitoring whether there is a situation where a person suddenly falls to the ground;

[0108] S4. Establish a pedestrian flow detection model for detecting and counting the number of people in the business hall;

[0109] S5. Monitor and judge whether there are difficulties during the process of customers handling business, and provide guidance to the customers.

[0110] S6. Transmit the data collected by the intelligent service robot to the central control analysis and display module. The central control analysis and display module analyzes the obtained data and displays it through the screen;

[0111] S7. According to the data analysis situation, dispatch the intelligent service robot through the intelligent scheduling module so that the intelligent service robot moves to the corresponding position to work.

[0112] In this solution, the central control analysis and display module will record the position of the intelligent service robot in real time. The work uniform detection model, personnel smoking model, personnel falling detection model, and pedestrian flow detection model are all carried on the environmental intelligent perception module. The image data captured by the intelligent service robot through the camera can be transmitted to the environmental intelligent perception module. The environmental intelligent perception module will analyze the collected images and transmit the analysis results to the central control analysis and display module. The central control analysis and display module will transmit the corresponding position data to the intelligent scheduling module. The intelligent scheduling module will dispatch the idle intelligent service robots so that they move to the corresponding positions for processing.

[0113] Further, step S1 includes the following steps:

[0114] S11. Collect a large amount of image data including wearing work uniforms and not wearing work uniforms, annotate the images, mark the specific positions of the work uniforms, and perform standardization processing on the images. Then, train the model according to the standardized images to form a work uniform detection model;

[0115] S12. The camera of the intelligent service robot captures the surrounding environment and uploads the collected images to the environmental intelligent perception module. The tooling detection model carried by the environmental intelligent perception module extracts local features in the images through a multi-layer convolutional neural network to obtain a feature map. The specific formula is as follows:

[0116]

[0117] Among them, I(x+i, y+j) is the input image pixel value, K(i, j) is the convolutional kernel, f(x, y) is the feature output, and i and j are spatial coordinates;

[0118] S13. The Region Proposal Network (RPN) generates candidate bounding boxes in the areas of the feature map that may contain tooling, and calculates the object scores and coordinate offsets of each candidate bounding box through a loss function. The calculation formula of the loss function is as follows:

[0119] L = L cls + λ·L reg

[0120] Among them, L cls represents the classification loss, which is used to calculate the object score, and L reg represents the regression loss, which is used to calculate the coordinate offset;

[0121] S14. The classification loss L cls evaluates whether the object score is accurate by using the cross-entropy loss. The specific formula is as follows:

[0122]

[0123] S15. The classification loss L reg uses the Smooth L1 loss to optimize the position of the bounding box, which is represented by the following formula:

[0124]

[0125] S16. Perform non-maximum suppression on the obtained results to remove overlapping bounding boxes. First, calculate the intersection over union (IoU) of two bounding boxes A and B to measure the overlapping degree of these two bounding boxes:

[0126]

[0127] Then, if the IoU of the two bounding boxes is greater than the set threshold, keep the bounding box with the higher object score and remove the bounding box with the lower object score, thereby giving the specific position of the tooling in the image in the form of a bounding box.

[0128] Among them, smooth L1 (x) is:

[0129]

[0130] Specifically, the process of step S11 can be achieved through the following:

[0131] Collect image data including the situations of wearing and not wearing work equipment. These images can come from multiple different sources, such as factory surveillance videos, on-site shootings, or simulated environments. To ensure data diversity, including different working environments, different work equipment styles, different worker postures, and different lighting conditions, this helps the model learn more extensive and robust features. And a large amount of image data needs to be collected because deep learning models usually require a large amount of data to learn complex patterns.

[0132] Then, each image needs to be annotated, including the wearing situation (wearing / not wearing) of the work equipment and the specific position of the work equipment. The position can be annotated by a bounding box, which is a rectangular area around the work equipment. Professional image annotation tools or software are needed to assist in the annotation process. These tools usually provide user-friendly interfaces and efficient work processes to ensure the accuracy of annotation, because incorrect annotation will cause the model to learn incorrect patterns, and it may require multiple people to conduct cross-validation and correction.

[0133] Adjust all images to a unified size, which helps reduce the computational amount and improve the training efficiency of the model. Then, normalize the images, scale the pixel values to the interval [0, 1] or [-1, 1] to help the model converge faster. After that, convert the images from the BGR (blue, green, red) color space to the RGB (red, green, blue) color space, which is the standard input for most deep learning models, and apply data augmentation techniques such as rotation, flipping, scaling, cropping, color jittering, etc. to increase the diversity of training data and improve the generalization ability of the model.

[0134] Among them, RPN is a network that quickly generates candidate regions (usually bounding boxes). These regions are considered possible target positions. For each candidate box, RPN calculates two values:

[0135] Object score: indicating whether the candidate box contains an object (for example, whether the work clothes are worn);

[0136] Coordinate offset: providing the precise position relative to the original candidate box, used to adjust the position and size of the candidate box to more accurately match the position and size of the real target;

[0137] Through the non-maximum suppression (NMS) algorithm, overlapping bounding boxes can be removed, and only the bounding box most likely to contain the work equipment is retained.

[0138] Furthermore, step S2 includes the following steps:

[0139] S21. Use a human pose estimation model such as OpenPose to obtain the position coordinates (x, y) of the key points of the human body (such as the mouth and hands), and minimize the position error through a loss function:

[0140]

[0141] Among them, y i represents the true key point coordinates, represents the predicted key point coordinates;

[0142] S22. For smoking action recognition, use 3D-CNN to extract features in the time and space dimensions. The 3D convolution calculation formula is as follows:

[0143]

[0144] Among them, I(x + i, y + j, z + k) represents the input frame, and K(i, j, k) represents the 3D convolution kernel;

[0145] S23. Perform temporal analysis through a long short-term memory network, and update the state specifically through the following formula:

[0146] Forget gate: f t = σ(W f ·[h t-1 , x t + b f )

[0147] Among them, W f represents the forget gate weight matrix, h t-1 represents the short-term memory, x t represents the current input, b f represents the bias term, and σ represents the Sigmoid function (outputs a probability value between 0 and 1);

[0148] Input gate: i t = σ(W i ·[h t-1 , x t + b i )

[0149]

[0150] Among them, represents the candidate cell state (new memory content), and the tanh function: generates a new feature representation between -1 and 1;

[0151] Cell state update:

[0152] Among them, ⊙ represents element-wise multiplication, Ct-1 Represents the cell state at the previous moment;

[0153] Output gate: o t = σ(W o ·[h t-1 , x t +b o )

[0154] h t = o t ⊙ tanh(C t )

[0155] Among them, h t Represents the feature vector for classification, used for classifying smoking behavior.

[0156] The staff smoking detection model focuses on identifying whether there is anyone smoking in the work business hall. This model can use deep learning technology to identify by training a large number of images containing cigarettes. The model analyzes the images in the real-time video stream, directly detects the features of the cigarettes, and if cigarettes are identified, it can give a voice alarm through an intelligent service robot.

[0157] Furthermore, step S3 includes the following steps:

[0158] S31. Detect human key points through the pose estimation model;

[0159] S32. Calculate the distance change of the key points:

[0160]

[0161] S33. Calculate the angle change of the key points:

[0162]

[0163] Among them, A and B are respectively vectors formed by adjacent key points;

[0164] S34. When a specific angle (such as the angle of the upper body) is lower than the set threshold, and the distance meets the falling-down feature (such as the vertical distance between the head and the ground), it is determined to have fallen down, and a support vector machine (SVM) classifier is used for determination:

[0165]

[0166] Among them, w is the weight, x i is the input feature vector, and y i is the label.

[0167] The staff falling detection model is mainly constructed by using the OpenPose model. This model collects a large amount of image data containing normal postures and falling postures during the training phase. During real-time monitoring, the model identifies the key points of the staff (head, shoulders, waist, knees, etc.), especially the center point A of the shoulder connection line and the midpoint B of the foot connection line. After connecting points A and B, it calculates the slope. If the slope exceeds the set threshold, the model determines that the person has fallen.

[0168] Further, step S4 includes the following steps:

[0169] S41. Conduct object detection: Generate a bounding box through global image division and object detection. The bounding box regression formula is as follows:

[0170] Center point coordinate regression:

[0171] t x =σ(p x )+c x

[0172] t y =σ(p y )+c y

[0173] Among them, p x and p y are prediction parameters, representing the center point offset, and c x and c y are the upper left coordinates of the cell;

[0174] Width and height regression:

[0175]

[0176] Among them, p w and p h are prediction parameters, representing the width and height of the prior box, and b w and b h represent the scaling factors of width and height;

[0177] S42. Conduct multi-object tracking, and perform state prediction and update through Kalman filtering. The prediction state equation is as follows:

[0178] x t|t-1 =F·x t-1|t-1 +B·u t-1

[0179] Among them, x t|t-1 represents the predicted state at the current moment, F is the state transition matrix, x t-1|t-1 represents the state estimate at the previous moment, B represents the control matrix, and u t-1 represents the control input;

[0180] Update equation:

[0181] x t|t = x t|t-1 + K t (z t - H·x t|t-1 )

[0182] Wherein, x t|t represents the update state at the current moment, K t is the Kalman gain, H is the observation matrix, and z t represents the observed value at the current moment;

[0183] S43. Use the Hungarian algorithm for matching to perform optimal matching between the detection box and the tracking box, and define the Euclidean distance matrix between the targets:

[0184]

[0185] Wherein, p i and q j represent the center coordinates of different targets, and the association between detection and tracking is achieved through the Hungarian algorithm.

[0186] Among them, the bounding box is the final output rectangular box in the object detection model, used to locate the target object in the image. The bounding box is jointly generated by the cell and the prior box: the center point coordinates are calculated based on the offset of the cell; the width and height are calculated based on the scaling factor of the prior box. Cell: Provides a reference position for the center point of the bounding box; Prior box: Provides the initial size of the width and height of the bounding box; Bounding box: Is the final output, adjusted by the cell and the prior box through the regression parameters.

[0187] Kalman filtering is a recursive algorithm that estimates the state of the target (such as position, speed, etc.) by combining prediction and observation information, and is divided into two main steps: prediction and update;

[0188] Prediction: Predict the state at the current moment based on the historical state and motion model of the target;

[0189] Update: Combine the observation data to correct the prediction result and obtain a more accurate state estimate;

[0190] Thus, through the above method, the state of the target can be effectively estimated and used for data association and state update in multi-object tracking.

[0191] The Hungarian algorithm is a bipartite graph matching algorithm used to find the optimal matching between two sets of objects, so that the total cost of the matching is minimized. In this solution, the Hungarian algorithm is used to associate the detection box with the tracking box to ensure that each detection box is matched with the most suitable tracking box

[0192] Further, step S5 includes the following steps:

[0193] S51. The intelligent service robot collects the actions of the customer during business handling, the voice prompts of the business terminal, and the prompts on the business terminal screen, and uses machine learning algorithms to determine whether the customer has performed the correct operation;

[0194] S52. When it is detected that the customer's actions do not meet the requirements of the current operation step, the intelligent service robot will trigger a feedback mechanism to prompt the customer to correct the actions;

[0195] S54. Connect the power-on / off hardware module to the business device for remotely controlling the power-on and power-off of the business device.

[0196] Taking the payment business as an example, the intelligent service robot in this solution captures the actions of the customer and the interface of the payment terminal through a camera, and analyzes whether the customer has encountered an operation error or is stuck at a certain step through image recognition technology, such as screen prompts, error messages, etc.; collects the voice prompts of the payment terminal, combines speech recognition technology to convert the voice into text, and recognizes the operation prompts or error prompts given by the current system; analyzes whether the customer understands the operation prompts and whether the correct steps have been executed by capturing the customer's actions (such as clicks, gestures) and combining machine learning algorithms.

[0197] Specifically, the above process mainly relies on image processing, gesture recognition, and machine learning algorithms to achieve intelligent judgment and feedback, specifically including the following:

[0198] 1. Customer action capture and recognition:

[0199] Action capture: The camera mounted on the intelligent service robot monitors the operations of the customer on the payment device in real time and captures the customer's hand movements and gestures;

[0200] Gesture recognition: The image recognition model classifies the captured image data for gesture classification to recognize specific actions (such as clicks, swipes, stays, placements, etc.). Common gesture classifications include "click on the screen", "long press", "double click", etc. Optionally, the image recognition model can be mounted in the central control analysis and display module;

[0201] Feature extraction: Extract features from the captured images to recognize the hand position, action direction, and strength, and judge the intention of the gesture.

[0202] 2. Action and operation process matching:

[0203] Step Identification and Comparison: The system first determines the steps of the current operation process from the current interface and the customer's operation prompts (such as "click on the payment method" or "enter the amount"), and then matches the identified customer actions with the correct steps;

[0204] Judgment of Action Validity: If the current step prompt requires the customer to click on a certain position or enter information, and the system detects that the customer's gesture does not conform to this step (for example, the gesture is a swipe or the click position is incorrect), then the action is judged as an incorrect operation;

[0205] Process Status Tracking: Throughout the operation process, the system continuously tracks the completion status of each step to ensure that the customer proceeds in the order of the steps. If a certain step is not completed or skipped, it may indicate that the customer does not understand the prompt for that step;

[0206] In this process, multi-modal information fusion technology is adopted to fuse the collected visual, voice, and interface information. Through natural language processing and computer vision technologies, the specific reasons for the user getting stuck at a certain operation step are analyzed. For example, whether the content prompted by the interface matches the customer's operation, and whether the voice prompt has been correctly understood.

[0207] 3. Analysis and Judgment of Machine Learning Model:

[0208] Learning of Error Types: Based on historical data, the system uses a machine learning model to identify common operation error types (such as repeatedly clicking, staying at a certain operation for a long time) and features, and judges whether the customer is encountering operation difficulties. For example, it judges whether there is a misoperation through the customer's gesture, click count, or position deviation;

[0209] State Prediction and Confusion Detection: Through comprehensive analysis of the customer's facial expression, eye gaze direction, and hand movements, it is judged whether the customer understands the current step. For example, when the customer stays at a certain interface for a long time or repeats the same action continuously, the machine learning model infers that the customer may be confused at this step.

[0210] 4. Real-time Feedback and Guidance Generation:

[0211] Feedback on Operation Errors: If the system detects that the customer's action does not meet the requirements of the current operation step, the intelligent service robot will trigger a feedback mechanism to prompt the customer to correct the action through voice or the screen mounted on the intelligent service robot. For example, when the customer accidentally clicks "Cancel" instead of "Confirm", the robot will prompt an operation error;

[0212] Understanding Evaluation and Guidance: When the customer's actions show that they do not understand the prompt or misunderstand the operation (such as frequently going back to the previous step or showing a confused expression), the robot can give instructions to guide the customer to understand the step again. For example, the system may prompt "Please click the confirmation button in the upper right corner" to further clarify the operation method;

[0213] Personalized prompt optimization: The system records the confusing steps that are prone to occur during the customer's operation process and gives early reminders during subsequent operations. For example, if the customer repeatedly enters the amount incorrectly, the system provides a special prompt at this step, which can be given through voice prompts and other means.

[0214] 5. Data accumulation and model iteration:

[0215] During the operation process, the system records the customer's operation steps, the number of corrections, and the confusing points to optimize the accuracy of the machine learning model. By continuously accumulating the customer's operation data, the model can better predict the operation difficulties that the customer may encounter and achieve personalized dynamic prompts.

[0216] Connecting the power-on and power-off module to the device to remotely control the power-on and power-off of the device is an existing technology that is easy for those skilled in the art to think of and will not be elaborated here.

[0217] Furthermore, step S5 further includes the following steps:

[0218] S54. Establish a troubleshooting model;

[0219] S55. Collect the images displayed on the business device interface and judge the problems existing in the business device;

[0220] S56. Give feedback to the customer according to the judgment result.

[0221] The process of establishing the troubleshooting model and judging the customer's problems can be divided into several steps, mainly including the construction of the fault knowledge base, the dual analysis method driven by rules and data, anomaly detection and fault matching, and intelligent feedback generation, which are mainly used to judge the possible problems of the device and give corresponding guidance to the customer. The following is the specific process:

[0222] 1. Construction of the fault knowledge base:

[0223] Problem and solution collection: During the system design phase, common fault types (such as network faults, payment failures, interface freezes, unsupported payment methods, etc.) and corresponding solutions are preset to form an initial fault knowledge base.

[0224] Problem label classification: Each fault is classified into different labels (such as "network error", "operation error", "payment failure", etc.) for quick matching.

[0225] Dynamic expansion and learning: During the system operation process, the newly added customer problems and solutions are stored in the knowledge base, gradually expanding the fault types and increasing the judgment accuracy. For example, if the system detects the "unsupported payment method" type of problem multiple times, it will automatically generate corresponding prompts and guidance.

[0226] 2. Dual analysis method of rule-driven and data-driven:

[0227] Rule-driven: The troubleshooting model judges the problems encountered by customers based on preset rules. For example, when the interface of the payment device shows "Network connection failed" or it is detected that the payment request fails to be sent to the server, the system can directly judge it as a network failure. The rule-driven method is applicable to known and easily identifiable fault types.

[0228] Data-driven: For complex or uncommon problems, the system relies on historical data and machine learning models to analyze the patterns of customer operations and judge possible fault causes. For example, by analyzing data such as the stagnation time and click frequency of customers during the payment process, the system identifies abnormal situations during operation, such as device interface freezing.

[0229] 3. Multimodal information fusion and anomaly detection:

[0230] Fusion of vision, voice, and interface prompts: The system fuses multimodal information such as visually captured customer gestures, voice prompts, and interface display content, and makes fault judgments through cross-modal comparison. For example, if the system identifies that the customer clicks the "Confirm" button, but the interface prompts "Operation timed out", the system may judge that there are network or system response problems with the device.

[0231] Anomaly behavior detection: The system judges whether the customer's operations are normal by analyzing anomalies in the customer's operation behaviors, such as repeated clicks, long stays, or clicks in the wrong position. For example, if the system detects that the customer clicks "Confirm" multiple times before the step of entering the amount, it identifies that the customer may misunderstand the operation process or encounter system freezing.

[0232] 4. Fault matching and intelligent reasoning:

[0233] Problem feature matching: Compare the detected customer operation features with the problem features in the fault knowledge base. For example, if the customer repeatedly tries to enter the same amount, but the interface prompts "Insufficient balance", the system matches this feature with the "Insufficient balance" type of fault.

[0234] Intelligent reasoning and fault judgment: When the customer operation features are not directly matched in the knowledge base, the system will use AI reasoning algorithms and combine anomalies from different information sources to predict possible faults. For example, if the customer's click operation does not trigger any interface feedback and the voice prompt is unresponsive, the system may infer abnormal device response or interface freezing.

[0235] 5. Problem classification and priority judgment:

[0236] Problem Classification: Based on the nature of the fault, the system classifies problems into operational faults (such as improper operation), systematic faults (such as network faults, payment interface faults), and user understanding problems (such as customer misunderstanding of prompts).

[0237] Priority Sorting: For multiple possible problems, the system will first provide the guidance that is most helpful to the customer. For example, network connection problems take precedence over operation guides, and payment interface faults take precedence over incorrect amount input errors.

[0238] 6. Intelligent Feedback Generation and Dynamic Guidance:

[0239] Immediate Feedback Generation: Based on the judged cause of the fault, the troubleshooting model generates personalized prompts. For example, if it is detected that the customer has misunderstood the prompt information, the system will provide clear operation guides and can tell the customer the specific operations through preset voice or other means.

[0240] Dynamic Guidance: The system continuously monitors the customer's operation response after the fault prompt. If it is detected that the problem is not solved, further guidance or additional prompts will be provided. For example, if the customer is still stuck on the payment interface after the prompt "Check network connection", the system will prompt the customer to restart the device or seek manual help.

[0241] Example: Customer Network Fault Judgment Process:

[0242] The system detects that after the customer clicks the "Confirm Payment" button, the interface does not respond and the interface shows a "Network connection failed" prompt;

[0243] The troubleshooting model matches this feature with the "Network fault" label;

[0244] If the system determines that the fault feature exactly matches the network problem, it will feedback to the customer "Network connection interrupted, please try again later";

[0245] If the problem is not solved, the system will trigger further feedback, such as "Try to reconnect to Wi-Fi or contact the staff".

[0246] 7. Continuous Optimization and Model Self-Learning:

[0247] Self-Learning of the Troubleshooting Model: The system records the customer's fault situations and solution feedbacks during actual operations, continuously optimizes the knowledge base and judgment rules, and improves the accuracy of the model.

[0248] Model Training and Iterative Update: Based on the newly accumulated data, the model is updated regularly to enhance its adaptability to new problems and improve the intelligence level of fault detection.

[0249] Furthermore, step S6 includes the following steps:

[0250] The data collected by the intelligent service robot is uploaded to the central control analysis and display module;

[0251] The collected data is displayed in real time on the screen;

[0252] The collected data is automatically generated into a report.

[0253] Through the above methods, the following aspects can be specifically achieved:

[0254] Business hall equipment monitoring: The intelligent service robot can conduct inspections and information collection in the business hall, and upload the operation status to the system. The system realizes the evaluation of the equipment operation status and displays it through a large screen. For equipment with abnormalities, the abnormal information is pushed to the display interface for timely reminder;

[0255] Mobile terminal operation monitoring: Real-time display of the operation status, work content, and completed work indicators of each intelligent service robot, such as the number of users received, the number of customers self-served, the number of customer appointments, the number of abnormalities found, the number of charging times, etc.;

[0256] Security monitoring: Actively identify and give early warnings and alarms for illegal operation behaviors in the business hall (such as work clothing detection, night patrol alarm, personnel smoking detection, personnel falling detection, etc.), so as to make the operations in the business hall more standardized and the supervision of the business hall more intelligent and automated; among them, the night patrol alarm is realized by the camera carried by the intelligent service robot. This camera is equipped with an infrared night vision function and has a fill light effect when the brightness is insufficient. When a person is detected at night, the intelligent service robot can give a voice prompt "You have entered the monitored area, please leave immediately", and take relevant photos and upload them to the server equipped with this system for subsequent investigation of the identity of the person.

[0257] Further, step S7 includes the following steps:

[0258] S71. According to the data of the surrounding environment collected by the intelligent service robot, the idle intelligent service robots are scheduled to the designated positions through the intelligent scheduling module;

[0259] S72. The route planned by the intelligent scheduling module is transmitted to the intelligent service robot, and the intelligent service robot can avoid obstacles according to the situation of nearby obstacles.

[0260] Specifically, the intelligent scheduling module in the intelligent navigator can perform global path planning, that is, determine the overall path from the starting point to the ending point for the intelligent service robot. The path planning module can enable the intelligent service robot to perform autonomous obstacle avoidance according to the situation of nearby obstacles observed, so as to achieve local path planning, and specifically can be implemented by using the dynamic window algorithm.

[0261] Taking the situation when there are many customers in the business hall as an example, the following scheduling method is adopted in this solution:

[0262] 1. Camera data collection

[0263] The cameras installed on the intelligent service robot are used to continuously and real-time capture the scenes in the business hall area. The image data transmitted back by each camera will be used as input and sent to the environmental intelligent perception module, and then the image data will be analyzed and input into the central control analysis and display module.

[0264] 2. People flow detection and counting

[0265] Object detection and object tracking are the key technologies for people flow analysis. Deep learning algorithms are mainly used to identify individuals in the crowd and track their movement paths.

[0266] Steps:

[0267] Background modeling: Establish a background model of the scene through historical data to distinguish the background from the foreground (dynamic objects);

[0268] Object detection: Use the object detection algorithm of deep learning, Faster R-CNN, to process the video frames in real time, identify and label individuals in the crowd;

[0269] Object tracking: After detecting each person, the object tracking algorithm DeepSORT can be used to associate the same object in consecutive frames to avoid double counting or missing counting.

[0270] 3. People flow density estimation

[0271] For the gathering situation of the crowd in the business hall, in addition to accurate counting, people flow density estimation can also be adopted. The people flow density in a certain area can be estimated through the density estimation algorithm CSRNet based on deep learning, without the need to identify individuals one by one.

[0272] 4. Congestion detection and analysis

[0273] By real-time analyzing the people flow density in different areas, calculate whether there is a congestion phenomenon in this area. According to this congestion index, early warnings can be issued or diversion can be guided.

[0274] 5. Intelligent guidance and diversion

[0275] Once the system detects a congestion trend in a certain area, or the business load of the VTM machines and counters is too high, the system can automatically enable the intelligent guidance and diversion strategy. The specific methods include:

[0276] · Use the voice function of the intelligent service robot to prompt customers to go to other areas.

[0277] · Guide customers to idle self-service devices to handle business through intelligent service robots.

[0278] 6. Integration of business handling data

[0279] Preferably, the business handling systems in the business hall (such as VTM machines and manual counters) can also upload the current business processing situations (such as processing speed and queuing number) to the central control analysis and display module. Combining the customer flow data and business data, the system can analyze the load situation of each business channel and further adjust the guidance strategy.

[0280] 7. Optimization of dynamic diversion

[0281] The system can combine historical data analysis and real-time data, and use optimization algorithms (such as queuing theory or reinforcement learning) to optimize resource allocation and personnel guidance strategies. For the prediction of queuing time, a model based on Markov chain can be used to predict future queuing situations.

[0282] Under normal circumstances, each intelligent service robot is deployed in each area (business handling area, exhibition hall introduction area, payment area, customer waiting area). The intelligent navigator is located at the front desk of the business hall, mainly responsible for greeting and reception. Other robots have different roles and are responsible for different tasks according to the area situation.

[0283] The scheduling methods in this embodiment are divided into three types:

[0284] Case 1: The intelligent navigator greets and receives customers, asks about customer needs, and plans and schedules each intelligent service robot through the intelligent scheduling module. For example, the intelligent service robot is scheduled to the front desk position to serve customers;

[0285] Case 2: During working hours, the intelligent scheduling module schedules each robot to the designated service position at regular intervals. For example, during working hours, the intelligent service robot is scheduled to the corresponding service area, and during off-duty hours, the intelligent service robot is scheduled to a position convenient for charging;

[0286] Case 3: The environmental intelligent perception module analyzes and obtains environmental abnormal information. Combining the established visual map coordinate relationship, the intelligent scheduling module schedules the intelligent service robot closest to the abnormal situation location to rush to handle it. At the same time, the intelligent scheduling module schedules the intelligent navigator to rush to the office position of the salesperson to notify him / her. Among them, the abnormal situation refers to situations such as someone smoking or someone falling down.

[0287] Preferably, the intelligent navigator in this embodiment can obtain customer requirements through voice communication, such as payment or business handling, and guide customers to state their true needs; the information collected by the intelligent service robot can be uploaded to the central control analysis and display module for real-time statistics. For services with an overly long handling time, they are regarded as overtime services, and statistics on overtime services are carried out for subsequent reminders to the staff in the business hall to optimize the business handling process.

[0288] The judgment criteria for overtime services can be as follows: combining existing statistical data, calculating the average value of the handling time for each service, comparing the service completion time with this average value, and if it is greater than the average value, it is counted as overtime.

[0289] Moreover, through this method, in the case where a customer has multiple inquiries during the handling process or a relatively high handling frequency at the same time every month, the customer's information can be automatically statistically analyzed and presented to the duty personnel in the business hall to contact the customer in advance to answer questions; and according to the consultation words provided by the customer, the knowledge base can be trained in subsequent work to enrich relevant knowledge answer materials.

[0290] Through the content described above, the situation in the unmanned business hall can be effectively monitored, and the customers and staff therein can be supervised. When the system detects an abnormal situation, the intelligent service robot can be dispatched through the intelligent scheduling module to move to the corresponding location for guidance or warning, thus improving the intelligence and automation of the robot team and enhancing the customer experience.

[0291] The above embodiments are only the preferred embodiments of the present invention. Those skilled in the art can obtain other embodiments from the above embodiments without creative efforts. Therefore, the present application protects not only the above embodiments, but also the scope consistent with the principles and features of the present application.

Claims

1. An unmanned business hall team system based on artificial intelligence, including multiple intelligent service robots, characterized in that, It also includes a path planning module connected inside the intelligent service robot, an intelligent scheduling module electrically connected to the path planning module, a central control analysis and display module electrically connected to the intelligent scheduling module, and an environmental intelligent perception module electrically connected to the central control analysis and display module. The central control analysis and display module is electrically connected to the intelligent control module; The intelligent scheduling module is used for scheduling the intelligent service robot. The central control analysis and display module is used for analyzing and displaying data. The environmental intelligent perception module is used for identifying and supervising the environment. The intelligent control module is used for guiding customers and controlling the start and stop of hardware devices.

2. A working method for the unmanned business hall team based on artificial intelligence, based on the unmanned business hall team system based on artificial intelligence as described in claim 1, characterized in that, Specifically, it includes the following steps: S1. Establish a tooling detection model for identifying whether the staff wears the tooling as required; S2. Establish a personnel smoking model for detecting and identifying whether there is a smoking situation of personnel; S3. Establish a personnel falling detection model for monitoring whether there is a situation where a person suddenly falls to the ground; S4. Establish a crowd flow detection model for detecting and counting the number of people in the business hall; S5. Monitor and judge whether there are difficulties during the process of customers handling business, and guide the customers; S6. Transmit the data collected by the intelligent service robot to the central control analysis and display module. The central control analysis and display module analyzes the obtained data and displays it on the screen; S7. According to the data analysis situation, schedule the intelligent service robot through the intelligent scheduling module to make the intelligent service robot move to the corresponding position for work.

3. The working method of the unmanned business hall team based on artificial intelligence according to claim 2, characterized in that, The step S1 includes the following steps: S11. Collect a large amount of image data including wearing tooling and not wearing tooling, annotate the images, mark the specific position of the tooling, and perform standardized processing on the images. Then, train the model according to the standardized images to form a tooling detection model; S12. The camera of the intelligent service robot will capture the surrounding environment. The model will extract local features in the image through a multi-layer convolutional neural network to obtain a feature map. The specific formula is as follows: Where, I(x+i, y+j) is the input image pixel value, K(i, j) is the convolutional kernel, f(x, y) is the feature output, and i and j are spatial coordinates; S13. Generate candidate bounding boxes in the area of the feature map that may contain tooling through RPN, and calculate the object score and coordinate offset of each candidate bounding box through a loss function. The calculation formula of the loss function is as follows: L = L cls + λ·L reg Among them, L cls represents the classification loss, which is used to calculate the target score, and L reg represents the regression loss, which is used to calculate the coordinate offset; S14. Classification Loss L cls The cross-entropy loss is used to evaluate whether the target score is accurate, and the specific formula is as follows: S15. Classification loss L reg Use Smooth L1 loss to optimize the position of the bounding box, which is expressed by the following formula: S16. Perform non-maximum suppression on the obtained results to remove overlapping bounding boxes. First, calculate the intersection over union of two bounding boxes A and B to measure the overlapping degree of these two bounding boxes: Then, if the IoU of the two bounding boxes is greater than the set threshold, retain the bounding box with the higher object score and remove the bounding box with the lower object score, so as to give the specific position of the tooling in the image in the form of a bounding box.

4. The working method of the unmanned business hall team based on artificial intelligence according to claim 2, wherein The step S2 includes the following steps: S21. Use a human pose estimation model to obtain the position coordinates (x, y) of the key points of the human body, and minimize the position error through a loss function: Among them, y i represents the true key point coordinates, and represents the predicted key point coordinates; S22. For the recognition of smoking actions, 3D-CNN is used to extract features in the time and space dimensions. The 3D convolution calculation formula is as follows: Among them, I(x + i, y + j, z + k) represents the input frame, and K(i, j, k) represents the 3D convolution kernel; S23. Perform temporal analysis through a long short-term memory network. The state is updated specifically through the following formula: Forgotten gate: f t = σ(W f ·[h t-1 , x t + b f ) Among them, W f represents the weight matrix, h t-1 represents the short-term memory, x t represents the current input, b f represents the bias term, and σ represents the Sigmoid function; Input gate: i t = σ(W i · [h t-1 , x t + b i ) Among them, represents the candidate cell state; Cell status update: where ⊙ represents element-wise multiplication, and C t-1 represents the cell state at the previous moment; Output gate: o t = σ(W o · [h t-1 , x t + b o ) h t = o t ⊙tanh(C t ) Among them, h t represents a feature vector for classification and is used for classifying smoking behavior.

5. The working method of the unmanned business hall team based on artificial intelligence according to claim 2, characterized in that, The step S3 includes the following steps: S31. Detect human key points through a pose estimation model; S32. Calculate the distance change of key points: S33. Calculate the angle change of key points: Among them, A and B are vectors formed by adjacent key points respectively; S34. When a specific angle is lower than the set threshold and the distance satisfies the falling-down feature, it is determined as a fall-down, and the determination is made through the following formula: Among them, w is the weight, x i is the input feature vector, y i is the label.

6. The working method of the unmanned business hall team based on artificial intelligence according to claim 2, characterized in that, The step S4 includes the following steps: S41. Perform object detection: Generate a bounding box through global image partitioning and object detection. The bounding box regression formula is as follows: Regression of the center point coordinates: t x = σ(p x ) + c x t y = σ(p y ) + c y Among them, p x and p y are prediction parameters, representing the center point offset, c x and c y are the upper left coordinates of the cell; Regression of the width and height: where p w and p h are prediction parameters, representing the width and height of the prior box, and b w and b h represent the scaling factors for the width and height; S42. Perform multi-object tracking. Perform state prediction and update through Kalman filtering. The prediction state equation is as follows: x t|t-1 = F·x t-1|t-1 + B·u t-1 where, x t|t-1 represents the predicted state at the current moment, F is the state transition matrix, and x t-1|t-1 represents the state estimate at the previous moment, B represents the control matrix, and u t-1 represents the control input; Update equation: x t|t = x t|t-1 + K t (z t - H·x t|t-1 ) where x t|t represents the update state at the current moment, K t is the Kalman gain, H is the observation matrix, and z t represents the observed value at the current moment; S43. Use the Hungarian algorithm for matching to perform optimal matching between the detection box and the tracking box, and define the Euclidean distance matrix between objects: Among them, p i and q j represent the central coordinates of different targets, and the association of detection and tracking is achieved through the Hungarian algorithm.

7. The working method of the unmanned business hall team based on artificial intelligence according to claim 2, characterized in that, The step S5 includes the following steps: S51. The intelligent service robot collects the actions of customers when handling business, the voice prompts of the business terminal, and the prompts on the screen of the business terminal, and uses machine learning algorithms to determine whether the customers have performed the correct operations; S52. When it is detected that the actions of the customers do not meet the requirements of the current operation steps, the intelligent service robot will trigger a feedback mechanism to prompt the customers to correct their actions; S53. Connect the power-on and power-off hardware module to the business device for remotely controlling the power-on and power-off of the business device.

8. The working method of the unmanned business hall team based on artificial intelligence according to claim 7, characterized in that, The step S5 also includes the following steps: S54. Establish a troubleshooting model; S55. Collect the images displayed on the interface of the business device and judge the problems existing in the business device; S56. Provide feedback to the customers according to the judgment results.

9. The working method of the unmanned business hall team based on artificial intelligence according to claim 2, characterized in that, The step S6 includes the following steps: S61. Upload the data collected by the intelligent service robot to the central control analysis and display module; S62. Real-time display the collected data on the screen; S63. Automatically generate reports for the collected data.

10. The working method of the unmanned business hall team based on artificial intelligence according to claim 2, characterized in that The step S7 includes the following steps: S71. According to the data of the surrounding environment collected by the intelligent service robot, the intelligent scheduling module schedules the idle intelligent service robots to the designated positions; S72. The route planned by the intelligent scheduling module will be transmitted to the intelligent service robot to make it move along the planned path; S73. During the movement of the intelligent service robot, it can autonomously avoid obstacles according to the observed nearby obstacle conditions.

Images