Service resource allocation method applied to service robot and service robot

Abstract

The embodiments of this specification relate to the field of artificial intelligence technology, and specifically disclose a service resource allocation method and a service robot applied to a service robot, wherein the method includes determining the location information of a target user to be handled based on the recognition result of a user in a designated area, and moving to the target user based on the location information; determining the current business handling needs of the target user based on the communication information with the target user; extracting at least one service device in the designated area that can realize the current business handling needs as an alternative service device; and based on the handling progress of the existing business handling needs of the alternative service device, selecting a service device that provides services to the target user from the alternative service devices, so that the selected service device provides services to the target user. Thus, the rationality of service resource allocation can be improved.

Description

Technical Field

[0001] The present specification relates to the field of artificial intelligence technology, and in particular, to a service resource allocation method applied to a service robot and a service robot. Background Art

[0002] At present, many bank branches are equipped with service robots. Service robots can provide services to users by communicating with them or showing them relevant guidance information. During the service process, it is very important for service robots to accurately provide users with the services they need to assist them in completing business transactions. Although current service robots can communicate with users in a simple way or provide simple business guidance, they are still unable to allocate service resources well to provide users with the services they need more accurately and efficiently. Summary of the invention

[0003] The purpose of the embodiments of this specification is to provide a service resource allocation method and a service robot applied to a service robot, which can improve the rationality of service resource allocation and the accuracy and efficiency of the service.

[0004] This specification provides a service resource allocation method for a service robot and a service robot that is implemented in the following manner:

[0005] A service resource allocation method applied to a service robot, the method comprising: determining the location information of a target user to be serviced based on the identification results of users in a specified area, and moving toward the target user based on the location information; determining the current service handling needs of the target user based on the communication information with the target user; extracting at least one service device in the specified area that can realize the current service handling needs as an alternative service device; and screening out service devices that provide services to the target user from the alternative service devices based on the processing progress of the existing business handling needs of the alternative service devices, so that the screened service devices provide services to the target user.

[0006] In some other embodiments of the method provided in this specification, the method of selecting service devices that provide services to the target user from the alternative service devices includes: for any existing business processing demand of the alternative service devices, obtaining the to-be-processed business type of the existing business processing demand and the to-be-processed node in the business type being processed, so as to determine the remaining processing time of the existing business processing demand according to the to-be-processed business type and the processing time of the to-be-processed node; based on the remaining processing time of the existing business processing demand of the alternative service devices, selecting service devices that provide services to the target user from the alternative service devices.

[0007] In some other embodiments of the method provided in this specification, a service device that provides services to the target user is used as the target service device, and the method further includes: determining the waiting time for the current business processing demand based on the remaining processing time of the existing business processing demand of the target service device; and feeding back the device identification information and waiting time of the target service device to the target user.

[0008] In some other embodiments of the method provided in the present specification, the service robot is equipped with a binocular camera, and the method further includes: using the binocular camera to collect information about the user to obtain a left image and a right image; and using the left image or the right image as a reference image; constructing a depth map based on the left image and the right image; back-projecting each pixel point in the reference image and the depth map into 3D coordinates in the camera coordinate system of the reference image to obtain 3D point cloud data; performing feature extraction on the reference image to obtain a feature map; and performing feature extraction on the 3D point cloud data to obtain a point cloud feature map; and locating the target user based on the feature map and the point cloud feature map.

[0009] In some other embodiments of the method provided in this specification, the positioning of the target user based on the feature map and the point cloud feature map includes: copying the feature map N times by column to obtain an intermediate feature map, where N represents the total number of points in the point cloud feature map; splicing the intermediate feature map with the point cloud feature map to obtain a spliced ​​feature map; normalizing the spliced ​​feature map to obtain an attention weight map; multiplying the attention weight map with corresponding elements of the point cloud feature map to obtain a fused feature map; and positioning the target user based on the fused feature map.

[0010] In some other embodiments of the method provided in this specification, the positioning of the target user based on the fused feature map includes: generating a two-dimensional candidate frame of the fused feature map using RPN; calculating a three-dimensional candidate frame corresponding to each two-dimensional candidate frame in combination with the depth map; wherein the size of the three-dimensional candidate frame is initialized based on the average size of each three-dimensional candidate frame in a training set; adjusting the three-dimensional candidate frame using a multi-task loss function to obtain a three-dimensional target frame of the target user; and positioning the target user according to the three-dimensional target frame.

[0011] On the other hand, the present specification also provides a service robot, which includes: a positioning module, which is used to determine the location information of a target user to be served based on the identification results of users in a specified area, so as to move toward the target user based on the location information; a demand determination module, which is used to determine the current business processing needs of the target user based on the communication information with the target user; an extraction module, which is used to extract at least one service device in the specified area that can realize the current business processing needs as an alternative service device; and a screening module, which is used to screen out service devices that provide services to the target user from the alternative service devices based on the processing progress of the existing business processing needs of the alternative service devices, so that the screened service devices can provide services to the target user.

[0012] In some other embodiments of the service robot provided in the present specification, the service robot is equipped with a binocular camera, which is used to collect information about the user to obtain a left image and a right image; and the left image or the right image is used as a reference image; the positioning module includes: a point cloud image construction unit, which is used to construct a depth image based on the left image and the right image; and in the camera coordinate system of the reference image, each pixel point in the reference image and the depth image is back-projected into 3D coordinates to obtain 3D point cloud data; a feature extraction unit, which is used to extract features from the reference image to obtain a feature image; and to extract features from the 3D point cloud data to obtain a point cloud feature image; a positioning unit, which is used to locate the target user based on the feature image and the point cloud feature image.

[0013] In some other embodiments of the service robot provided in this specification, the positioning module also includes a weight map construction unit and a fusion unit; the weight map construction unit is used to copy the feature map N times by column to obtain an intermediate feature map, wherein N represents the total number of points in the point cloud feature map; and to splice the intermediate feature map with the point cloud feature map to obtain a spliced ​​feature map; and to normalize the spliced ​​feature map to obtain an attention weight map; the fusion unit is used to multiply the attention weight map with the corresponding elements of the point cloud feature map to obtain a fused feature map; and the positioning unit is also used to locate the target user based on the fused feature map.

[0014] In some other embodiments of the service robot provided in this specification, the positioning module also includes a candidate frame determination unit and a target frame determination unit; the candidate frame determination unit is used to generate a two-dimensional candidate frame of the fused feature map using RPN; and calculate the three-dimensional candidate frame corresponding to each two-dimensional candidate frame in combination with the depth map; wherein the size of the three-dimensional candidate frame is initialized based on the average size of each three-dimensional candidate frame in a training set; the target frame determination unit is used to adjust the three-dimensional candidate frame using a multi-task loss function to obtain a three-dimensional target frame of the target user; the positioning unit is also used to locate the target user according to the three-dimensional target frame.

[0015] The service resource allocation method and service robot applied to the service robot provided by one or more embodiments of this specification can actively communicate with the user by accurately locating the user. And the user's business handling needs can be determined based on the communication information, so as to provide more accurate and effective business handling assistance to the user and improve the user experience. After determining the business handling needs, the service equipment that can realize the business handling needs can be further extracted as the alternative service equipment, and then the business volume and business processing progress of each alternative service equipment can be analyzed, and the service order and counter resources can be reasonably allocated. The service equipment that can provide services to the user more quickly and accurately can be selected from the alternative service equipment, thereby reducing the user's waiting time. At the same time, the efficiency and accuracy of the overall business handling of the business hall can also be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art description. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. In the drawings:

[0017] Figure 1 A flowchart of an embodiment of a service planning method for a service robot provided in this specification;

[0018] Figure 2 A schematic diagram of a feature extraction process in an embodiment provided in this specification;

[0019] Figure 3 A schematic diagram of a data fusion process in an embodiment provided in this specification;

[0020] Figure 4 A schematic diagram of a neural network regression and classification process in an embodiment provided in this specification;

[0021] Figure 5A schematic diagram of a three-dimensional target detection process in an embodiment provided in this specification;

[0022] Figure 6 This is a schematic diagram of the module structure of the service robot provided in this manual. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below in conjunction with the drawings in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of the specification, not all of the embodiments. Based on one or more embodiments of the specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the embodiments of this specification.

[0024] In a scenario example of this specification, the service robot can be a mobile intelligent terminal device configured in a bank business hall. The service robot can be configured with a collection module (such as a camera, a fingerprint collection module, a voice collection module, etc.), a display, a processor, a memory, a communication module, etc. The service robot can collect the user's facial, fingerprint, voice and other biometric information through the collection module. The processor of the service robot can be used to implement data processing such as user identity recognition, path planning, business planning, business processing, and scenario intelligent communication based on the collected information. The service robot can also establish a communication connection with the business system of the financial institution through the communication module. Accordingly, the service robot can also send the collected information to the business system so that the business system performs data processing such as user identity recognition, path planning, business planning, business processing, and scenario intelligent communication, and the service robot can receive the data processing results fed back by the business system. The service robot can display the data processing results to the user through a display, or it can feed back to the user in the form of voice. Correspondingly, the memory can store some basic data or implementation programs for the service robot to perform the above data processing.

[0025] For example, when a user enters a bank's business hall to handle business, the service robot can collect information about the user, locate the user, and greet the user. For example, the service robot can greet the target user, "Hello, how can I help you?" The target user can ask the service robot, "I want to buy financial products, can you introduce them to me?" The service robot can further ask the target user, "Our bank probably has... types of financial products, which one do you want to know more about?" Accordingly, the target user can reply, "Please show me XX types of financial products." Accordingly, the service robot can initially display or introduce information about XX types of financial products to the target user. After the display or introduction, the service robot can further reply to the user, "If you want to buy this type of financial product or want to learn more about it, you can go to counter A to handle it."

[0026] In this process, the service robot can extract the user's business handling needs from the communication information with the user. For example, in this scenario example, the service robot can extract that the target user's business handling needs are "purchase of financial products". Then, at least one service device in the specified area that can realize the current business handling needs of "purchase of financial products" can be extracted as an alternative service device. Service equipment can be, for example, a manual counter, an intelligent counter, etc. The existing business handling needs of each alternative service device and the progress of the existing business handling needs can be further analyzed to screen out the service devices that provide services to the target user from the alternative service devices based on the analysis results, and guide the user to the screened service device for business handling.

[0027] By accurately locating users, actively communicating with them, and determining their business needs based on the communication information, we can understand their needs in a timely and accurate manner, and provide more accurate and effective business assistance to users, thereby improving user experience. After determining the business needs, we can further extract service equipment that can meet the business needs as alternative service equipment, and then analyze the business volume and business processing progress of each alternative service equipment, reasonably allocate service order and counter resources, and select service equipment from the alternative service equipment that can provide services to users more quickly, reducing user waiting time, and at the same time improving the efficiency and accuracy of the overall business handling of the business hall.

[0028] Figure 1It is a flowchart of an embodiment of the business planning method applied to a service robot provided in this specification. Although this specification provides method operation steps or device structures as shown in the following embodiments or drawings, more or fewer operation steps or module units may be included in the method or device based on routine or no creative labor. In the steps or structures where there is no necessary causal relationship logically, the execution order of these steps or the module structure of the device is not limited to the execution order or module structure shown in the embodiments or drawings of this specification. When the method or module structure is applied to an actual device, server or terminal product, it can be executed sequentially or in parallel according to the method or module structure shown in the embodiments or drawings (for example, in an environment of parallel processors or multi-threaded processing, or even including distributed processing, an implementation environment of a server cluster). A specific implementation example is Figure 1 As shown, in one embodiment of a service planning method applied to a service robot provided in this specification, the method can be applied to a service robot, and the method may include the following steps:

[0029] S20: Determine the location information of the target user to be serviced according to the identification result of the user in the designated area, and move to the target user based on the location information.

[0030] The service robot can identify users in a designated area. The service robot can be configured with a collection module to collect the user's facial information, language information, etc. The service robot or the business system can be configured with algorithms for performing facial recognition, voice recognition, etc. The determination of the target user can be performed by the service robot or by a business system that establishes a communication connection with the service robot.

[0031] The designated area may be a bank's business handling location, a mobile business handling vehicle, etc. The target user may be a user who has entered the designated area but has not been assigned a service device. The service device may be a terminal device used to implement business handling in the designated area.

[0032] For example, if the designated area is a branch business hall of a bank, the bank's business hall is usually equipped with multiple service devices. Service equipment may include smart counters and manual counters. Of course, service equipment may also include service robots themselves. Smart counters or service robots can usually provide services to users automatically. For example, users can operate the smart counter interface to complete business processing. Manual counters are usually operated by business personnel to operate service equipment (such as computers) to complete business processing for users.

[0033] If a user enters a business hall, the service robot can perform facial recognition on the user to determine whether the user has been assigned a service device. If not, the user can be used as a target user. Of course, other biometric information of the user can also be identified to determine the user's identity information, and then determine whether the user has been assigned a service device. Timely detection of users who have entered the business hall but have not been assigned a service device can help users handle business more quickly and effectively, improving the user experience.

[0034] Alternatively, the target user may also be a user who has been assigned a service device in other implementations. If a user has been assigned a service device and wants to readjust the business handling needs and consult the service robot, the user may also be the target user.

[0035] Of course, inspired by the technical essence of the embodiments of this specification, technical personnel in the relevant field may have other changes in the implementation method for determining the target user, but as long as the functions and effects achieved are the same or similar, they should be covered within the scope of protection of this specification.

[0036] The service robot can also locate the target user's position to track and locate the target user, facilitate communication with the target user, and improve the target user's interactive experience.

[0037] For example, a service robot can be equipped with a binocular camera as the robot's eyes. The binocular camera is used to collect images of the area where the target user is located. The images collected by the two cameras can be referred to as left image I and right image II. l (u, v) and right figure I r (u, v), and calculate the corresponding depth map D(u, v).

[0038]

[0039] Among them, f U is the horizontal focal length of the left image, Y(u, v) is the disparity map between the left image and the right image, and b is the baseline of the binocular camera.

[0040] Then, the target user can be tracked and located. When detecting the target user based on the information collected by the binocular camera, the target detection model can be pre-trained to detect only people. This design can greatly improve the efficiency and accuracy of determining the target user, allowing the service robot to detect the user more quickly.

[0041] The left image or the right image can be used as the reference image for subsequent processing. If the left image is used as the reference image, each pixel (u, v) can be back-projected to 3D coordinates (x, y, z) in the camera coordinate system of the left image to obtain 3D point cloud data {(x (n) ,y(n) , z (n) )} N n＝1 .

[0042] (depth)z=D(u,v)

[0043]

[0044]

[0045] Where x, y, and z represent the depth coordinate, horizontal coordinate, and vertical coordinate of the 3D coordinates, respectively; n represents an arbitrary pixel point; N is the total number of pixels in the 3D point cloud; and c U 、c V is the position offset of the pixel coordinate origin relative to the center of the camera corresponding to the left image, f u is the vertical focal length.

[0046] like Figure 2 As shown in the figure, the neural network Faster-RCNN is used to extract features from the left image to obtain the feature map F(C, 1024). The PointNet++ network is used to extract features from the 3D point cloud data generated above to obtain the point cloud feature map F2(N, 64). Subsequent processing using the feature map and the point cloud feature map can greatly reduce the amount of data processing compared to directly using radar to collect point cloud data.

[0047] In some embodiments, data fusion based on an attention mechanism may be used to fuse the feature map F and the point cloud feature map F2.

[0048] like Figure 3 As shown, the specific data fusion process can be:

[0049] (a) Input feature map F(C, 1024) and point cloud feature map F2(N, 64).

[0050] (b) According to the total number of points N in the point cloud feature map F2, the feature map F(C, 1024) is copied N times by column to obtain the intermediate feature map F1(Nx1024).

[0051] (c) Concatenate F1 (Nx1024) and F2 together to form a concatenated feature map of size Nx1088.

[0052] (d) Through the MLP layer and the normalization function (sigmoid), the concatenated feature map is normalized to the interval [0,1] to obtain the attention weight map (Attention Map (Nx64)). The attention weight map is used to represent the attention of the neural network to each region. The higher the value, the higher the attention.

[0053] (e) Multiply the attention weight map with the corresponding element of the point cloud feature map F2 to obtain the Refined Feature (improved feature map); add the Refined Feature to the point cloud feature map F2 through the residual connection to obtain the fused feature map (Out Feature (Nx64)).

[0054] The fusion solution provided in the above embodiment adopts different mining methods for different data based on the color advantages of RGB data and the spatial advantages of point cloud data, so as to maximize the advantages of each data. The above fusion method obtains the attention weight map by means of the RGB feature map and the point cloud data feature map, and uses the attention weight map to guide the feature map extracted from the point cloud data so that the extracted feature map can better express the features. The attention weight map can well describe the relative relationship between different local structures and the contribution of each local structure to the recognition of characters. It can strengthen the information of characters in the point cloud and weaken other interference information, thereby improving the accuracy of character detection.

[0055] like Figure 4 As shown, the fused feature map can be further fed into a convolutional neural network for target classification and regression to obtain the position coordinates of the person in the three-dimensional space. In this embodiment, the network can be divided into two steps based on the three-dimensional target detection of the prior candidate box (such as Figure 5 3D candidate box generation and 3D target box regression. 3D candidate box generation is used to generate a series of 3D candidate boxes, and 3D target box regression is used to screen the 3D candidate boxes to further obtain accurate 3D target boxes.

[0056] refer to Figure 5 , FC1, FC2, FC3, and FC4 represent fully connected layers respectively. RPN (Region Proposal Network) can be used to generate two-dimensional candidate boxes. The input of RPN is an image of any size, and the output is a set of rectangular target candidate boxes, each of which corresponds to a probability value. Then, the 2D candidate boxes (two-dimensional candidate boxes) obtained above are used to initialize the 3D candidate boxes (three-dimensional candidate boxes) in combination with the depth map. For each 2D candidate box, the corresponding 3D candidate box is calculated. The 3D candidate box is represented by a 7-dimensional vector (x c ,y c ,z c ,l,w,h,θ). (x c ,y c ,z c) are the coordinates of the center point in the camera coordinate system. (l,w,h) represents the size of the 3D candidate box, and θ represents its rotation around the y axis by θ in the camera coordinate system (π,-π). For the size of the 3D candidate box, a simple statistics can be performed on the training set, and the 3D candidate box can be initialized with the estimated average size of the box. Usually, people have a general idea of ​​the size of familiar objects (such as cars, tables, etc.). Using the estimated average size of the box to initialize the 3D candidate box is better than directly using high-level feature maps to fit the size of the 3D box.

[0057] The initial value of the direction angle θ of all 3D candidate boxes is set to 0. The center point (x c ,y c ,z c )The initial value is calculated as shown in the following formula:

[0058] x c =z med *(c x -o x ) / f

[0059] y c =z med *(c y -o y ) / f

[0060] Where, f is the focal length of the camera, (o x ,o y ) is the main axis point, (c x ,c y ) is the center of the 2D candidate box, z c Equal to z med , z med It is the average depth value of the corresponding 2D candidate box in the depth map.

[0061] A complete 3D target box (3D target box) can be further output for each target object. The 3D target regression network refines the 3D candidate box according to the previously obtained feature map to obtain the final 3D target box. t = (Δx, Δy, Δz, Δl, Δw, Δh, Δθ) can be regressed to indirectly realize the regression of the 3D box, and then determine the difference between the target boxes. Among them, (Δx, Δy, Δz) is the offset of the center point of the 3D target box relative to the center point of the 3D candidate box, which is scale-invariant normalized. (Δw, Δh, Δl) is the offset of the size of the 3D target box relative to the size of the 3D candidate box, which is log-space shift normalized. In addition, Δθ is the offset of the deflection angle of the 3D target box relative to the deflection angle of the 3D candidate box, and Δθ is given by Δθ = θ-θ a , Δθ * =θ * -θa Calculated. Among them,

[0062] Δx=(xx a ) / w a ,Δy=(yy a ) / h a ,Δz=(zz a ) / l a ,

[0063] Δw=log(w / w a ),Δh=log(h / h a ),Δl=log(l / l a ),

[0064] Δx * =(x * -x a ) / w a ,Δy=(y * -y a ) / h a ,Δz=(z * -z a ) / l a ,

[0065] Δw=log(w * / w a ),Δh=log(h * / h a ),Δl=log(l * / l a ),

[0066] x, y, z, w, h, l represent the center coordinates of the 3D target box and its width, height, and length. a and x* represent the x-coordinates of the target box, candidate box, and ground-truth center points, respectively. The other variables y, z, w, h, and l are similar to x.

[0067] A multi-task loss function can be used to simultaneously predict the class label, 3D object box offset, and object orientation, as shown in the following formula.

[0068]

[0069] The superscripts p and r represent 3DRPN and 3DRCNN respectively, and the subscripts box and θ represent the loss functions of the 3D target box and orientation respectively. The category loss can be defined as the cross entropy loss, and the orientation loss and target box offset loss can be defined as the smooth L1 loss. Each loss is weighted according to its uncertainty.

[0070] Based on the above implementation mode, in some embodiments, the service robot can use a binocular camera to collect information from the user to obtain a left image and a right image; and use the left image or the right image as a reference image; construct a depth map based on the left image and the right image; back-project each pixel point in the reference image and the depth map to 3D coordinates in the camera coordinate system of the reference image to obtain 3D point cloud data; perform feature extraction on the reference image to obtain a feature map; and perform feature extraction on the 3D point cloud data to obtain a point cloud feature map; and locate the target user based on the feature map and the point cloud feature map.

[0071] In the process of locating the target user based on the feature map and the point cloud feature map, the feature map can be copied N times by column to obtain an intermediate feature map, where N represents the total number of points in the point cloud feature map; the intermediate feature map and the point cloud feature map are spliced ​​to obtain a spliced ​​feature map; the spliced ​​feature map is normalized to obtain an attention weight map; the attention weight map is multiplied by corresponding elements of the point cloud feature map to obtain a fused feature map, so as to locate the target user based on the fused feature map.

[0072] In the process of locating the target user based on the fused feature map, RPN can also be used to generate a two-dimensional candidate frame of the fused feature map; the three-dimensional candidate frame corresponding to each two-dimensional candidate frame is calculated in combination with the depth map; wherein the size of the three-dimensional candidate frame is initialized based on the average size of each three-dimensional candidate frame in the training set; the three-dimensional candidate frame is adjusted using a multi-task loss function to obtain the three-dimensional target frame of the target user; and the target user is located according to the three-dimensional target frame.

[0073] Of course, inspired by the technical essence of the embodiments of this specification, technicians in the relevant field may also come up with other modified implementation methods, but as long as the functions and effects achieved are the same or similar, they should be included in the protection scope of this specification.

[0074] For example, point cloud data can be projected onto a two-dimensional plane. This method does not directly process three-dimensional point cloud data, but projects the point cloud onto certain specific perspectives before processing, such as the front view and the bird's-eye view. At the same time, the image information from the camera is fused and features are extracted through a neural network to obtain a fused feature map. By combining data from these different perspectives, the cognitive task of point cloud data is achieved. However, this method simply splices different types of data and does not really take advantage of the different types of data.

[0075] Alternatively, the point cloud data can be processed first, and then RGB feature maps can be used to assist. The point cloud data is divided into voxels with spatial dependencies. This method introduces spatial dependencies into the point cloud data by segmenting the three-dimensional space, and then uses 3D convolution and other methods to process it. After obtaining the processed point cloud data, RPN is used with the assistance of the RGB map for classification and target box regression. However, the accuracy of this method depends on the fineness of the segmentation of the three-dimensional space, and the computational complexity of 3D convolution is also high.

[0076] After locating the target user's position in three-dimensional space, the path can be further planned to guide the displacement and direction of the robot's movement. The service robot can also locate the target user's position in real time based on the above method and adjust the displacement and direction of movement. The service robot can approach and communicate with the target user based on the determined displacement and direction of movement.

[0077] S22: Determine the current business handling needs of the target user based on the communication information with the target user.

[0078] The service robot may generate communication information during the communication process with the target user. The communication information may include voice communication information between the target user and the service robot, or may include information selected or input by the target user in the interface displayed by the service robot. Of course, other types of communication information may also be included, which are not limited here.

[0079] The service robot can use relevant algorithms such as situational intelligence to determine the current business handling needs of the target user based on the communication information with the target user. Business handling needs can be, for example, "opening an account", "depositing / withdrawing", "converting current accounts to fixed deposits", "purchasing financial products", etc. Of course, the service robot can also send the communication information with the target user to the business system, so that the business system can determine the current business handling needs of the target user and feedback to the service robot.

[0080] When constructing the relevant algorithm for determining the current business handling needs, the actual business situation of the bank can also be combined to extract business sensitive words to form a business sensitive word set. The relevant algorithm for determining the business handling needs is constructed in combination with the business sensitive word set, so that the current business handling needs of the user can be located more quickly and accurately from the communication information. Accordingly, in some embodiments, the current business handling needs of the target user can be determined based on the pre-constructed business sensitive word set.

[0081] For example, you can extract features from voice communication information, select appropriate recognition units, define a business-sensitive word sequence, and then use a hidden Markov model to build a voice recognition model for each unit. For each different word in the recognizer vocabulary, make a rough model, then perform Viterbi training before the model converges, and perform a few (up to 10) iterations of Baum-Welch training. Then, you can use the trained voice recognition model to identify the current business processing needs of the target user from the voice communication information.

[0082] Alternatively, the service robot may further determine the current business processing needs of the target user by combining the target user's business processing information obtained from other devices that communicate with it (such as a number-taking machine or a business processing intelligent terminal).

[0083] Alternatively, the service robot can also extract features from the collected visual and voice information, and then use the hidden Markov model to build a recommendation model. When users are unable to accurately describe their needs, the recommendation model can be used to accurately predict current business needs with the help of other relevant information of the target user and the needs of previous business transactions. The prediction model can feed back the prediction results to the input end, continuously iterate the model, and further enhance the accuracy and robustness of the model. In order to protect the privacy of users, multi-party secure computing and federated learning technology can be used when using data from other companies to ensure data security.

[0084] Inspired by the technical essence of the embodiments of this specification, technicians in the relevant field may also come up with other modified implementation methods, but as long as the functions and effects achieved are the same or similar, they should be included in the protection scope of this specification.

[0085] S24: extract at least one service device in the designated area that can meet the current business processing demand as a candidate service device.

[0086] The functions implemented by different service devices may vary. For example, some smart counters can only handle basic services such as deposits and withdrawals, such as ATMs. Some smart counters can handle services such as opening accounts and converting current accounts to fixed deposits. Some manual counters can handle multiple types of services, while some manual counters can only handle financial management and insurance services. Alternatively, some manual counters only handle services for VIP users.

[0087] The device identification information of each service device can be associated with the achievable business processing information, the targeted user range, etc. in advance and stored to form a device service set. After determining the current business processing needs of the target user, the service robot can extract at least one service device in the business hall that can realize the current business processing needs from the device service set as an alternative service device. If the current business processing need is "opening an account", the device identification information of the smart counter or manual counter in the business hall that can realize the account opening business can be extracted. Of course, the identity information of the target user can be further referenced to extract the service equipment that can realize the current business processing needs corresponding to the target user as an alternative service equipment.

[0088] Alternatively, if the current business handling demand is within the business scope that the service robot can handle, the service robot can be used as an alternative service device. The service robot can directly handle the current business handling demand without executing step S26. When the service robot cannot complete it, other service devices can be further extracted as alternative service devices, and step S26 can be executed for screening. Of course, the service robot can also guide users to prepare the relevant materials needed to meet the current business handling needs, so that users can save queuing time without sufficient preparation, thereby improving user satisfaction.

[0089] Inspired by the technical essence of the embodiments of this specification, technicians in the relevant field may also come up with other modified implementation methods, but as long as the functions and effects achieved are the same or similar, they should be included in the protection scope of this specification.

[0090] S26: Based on the processing progress of the existing business processing requirements of the candidate service devices, a service device that provides services to the target user is screened out from the candidate service devices, so that the screened service device provides services to the target user.

[0091] After determining the alternative service device, the service robot can obtain the existing business handling requirements of each alternative service device. The existing business handling requirements may refer to the business handling requirements of other users that have been assigned to the corresponding alternative service device before the current business handling requirements but have not yet been completed. The existing business handling requirements may include the business handling requirements of users that are being processed by the alternative service device, and may also include the business handling requirements of users who are waiting in line but have not yet started to be processed.

[0092] The service robot can further obtain the processing progress of the existing business processing requirements. The processing progress may include being processed, not yet processed, etc. In some embodiments, the remaining processing time of the existing business processing requirements can be used to represent the processing progress of the existing business processing requirements.

[0093] At least one business type involved in the existing business handling needs can be obtained. Business type is the type of banking business divided in the bank's business system. It can usually include deposits, withdrawals, account opening, financial management, insurance, etc. The business processing flows corresponding to the business processing of each business type are usually different, and there are usually multiple processing nodes in the business processing data flow of each business type. For any business type, the processing nodes it includes and the processing time of each processing node can be extracted. The processing time of each processing node can be determined by statistically analyzing the historical business processing data corresponding to the business type. The processing time required for each business type can be determined based on the processing time of each processing node of the business type.

[0094] For existing business processing requirements being processed, the business types involved in the existing business processing requirements can be extracted from the existing business processing requirements. Then, the business types to be processed in the existing business processing requirements and the nodes to be processed in the business types being processed can be obtained, and the remaining processing time of the existing business processing requirements can be determined based on the sum of the processing time of the business types to be processed and the nodes to be processed, so as to represent the processing progress of the existing business processing requirements.

[0095] For existing business handling needs that have not been processed in time, the business types involved can also be obtained. The business types of existing business handling needs that will be processed in the future have not yet started processing, so they are all pending business types. Based on the sum of the processing time of each pending business type involved in the existing business handling need, the processing time required to complete the business handling of the existing business handling need can be determined as the remaining processing time of the existing business handling need to represent the processing progress of the existing business handling need.

[0096] Accordingly, for any existing business processing demand of the alternative service device, the pending business type of the existing business processing demand and the pending node in the business type being processed can be obtained to determine the remaining processing time of the existing business processing demand according to the pending business type and the processing time of the pending node. Then, based on the remaining processing time of the existing business processing demand of the alternative service device, the service device that provides services to the target user can be screened out from the alternative service devices. In this way, the service device that can provide services to the target user the fastest can be screened out from the alternative service devices, so as to more reasonably allocate service resources in the business hall during the peak period of business processing, reduce the waiting time of users, and improve the user experience.

[0097] During peak business processing periods, after initially screening out the service equipment that provides services to the target user from the alternative service equipment, it is possible that the service equipment cannot provide services to the user in a timely manner. The service equipment that is screened out to provide services to the target user can be used as the target service equipment, and the waiting time required by the user can be determined based on the existing business processing needs that have not yet been processed by the target service equipment and the remaining processing time of the existing business processing needs that are being processed. The service robot can provide feedback to the user on the required waiting time. Further providing the user with the waiting time allows the user to freely control the time. If the waiting time is short, the user can wait for a while in the designated area. If the waiting time is long, the user can use this part of time to handle other things, thereby improving the user experience.

[0098] As time goes by, the service robot can also dynamically adjust the service equipment and waiting time provided to the target user based on the existing business processing needs of each alternative service device and the real-time progress of the existing business processing needs, and provide timely feedback to the target user.

[0099] The types of business operations that can be realized by some service devices may be partially overlapping. For example, a certain smart counter A can realize account opening business operations, and most manual counters can also realize account opening business operations, but the smart counter A cannot realize financial management business operations, while some manual counters can realize financial management business operations. Therefore, the business operation types corresponding to the existing business operation requirements of each alternative service device corresponding to the current business operation requirements may also be different. For example, if the alternative service devices are smart counter A and manual counter B, both smart counter A and manual counter B can realize account opening business operations, but smart counter A cannot realize financial management business operations, and manual counter B can realize financial management business operations.

[0100] When selecting service devices from the candidate service devices, in addition to analyzing the existing business handling needs corresponding to each candidate service device and the handling progress of the existing business handling needs, the existing business handling needs of other service devices with existing business handling needs can also be analyzed to integrate multiple information, perform dynamic planning, and more reasonably select service devices that provide services to the target users. Through dynamic planning, the resource utilization of each service device in the business hall can be adjusted in real time, the rationality of service order allocation can be improved, and service devices can be provided to users more quickly and accurately, reducing user waiting time.

[0101] Alternatively, the service robot can also collect information on whether there are users who are currently handling business at each alternative service device in the specified area. If not, the alternative service device does not have any existing business handling needs to be handled, that is, the handling progress of the existing business handling needs of the alternative service device can be considered to be empty. At this time, the device identification information of the alternative service device can be directly displayed to the target user or the target user can be replied to by voice, so that the target user can go to the alternative service device to handle business. During the off-peak period of business handling, a large number of service devices may be idle. At this time, based on the collection results of whether there are users who are currently handling business in front of each service device, the service equipment is allocated, and the service equipment that can meet their business handling needs can be fed back to the user more quickly. At the same time, there is no need to execute complex resource allocation algorithms, which can reduce the data processing volume of the service robot or business system.

[0102] In the solution provided by the above embodiment, the service robot can accurately locate the user and actively communicate with the user in need. And the user's business handling needs can be determined based on the communication information, so as to provide more accurate and effective business handling assistance to the user and improve the user experience. After determining the business handling needs, the service equipment that can realize the business handling needs can be further extracted as the alternative service equipment, and then the business volume and business processing progress of each alternative service equipment can be analyzed, and the service order and counter resources can be reasonably allocated. The service equipment that can provide services to the user more quickly can be selected from the alternative service equipment, thereby reducing the user's waiting time. At the same time, it can also improve the overall business handling efficiency and service accuracy of the business hall.

[0103] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. For details, please refer to the description of the above-mentioned related processing related embodiments, and no further description is given here.

[0104] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0105] Figure 6 A schematic diagram of the module structure of a service robot provided in the manual is shown in FIG. Figure 6 As shown, the service robot may include the following modules.

[0106] The positioning module 602 may be used to determine the location information of the target user to be serviced according to the identification result of the user in the designated area, so as to move toward the target user based on the location information.

[0107] The demand determination module 604 may be used to determine the current business handling demand of the target user based on the communication information with the target user.

[0108] The extraction module 606 may be used to extract at least one service device in the designated area that can meet the current business processing requirement as a candidate service device.

[0109] The screening module 608 may be used to screen out service devices that provide services to the target user from the candidate service devices based on the processing progress of the existing business processing requirements of the candidate service devices, so that the screened service devices provide services to the target user.

[0110] In some other embodiments, the service robot is equipped with a binocular camera, which can be used to collect information from the user to obtain a left image and a right image; and the left image or the right image is used as a reference image.

[0111] The positioning module may include:

[0112] The point cloud image construction unit can be used to construct a depth map based on the left image and the right image; and back-project each pixel point in the reference image and the depth map into 3D coordinates in the camera coordinate system of the reference image to obtain 3D point cloud data.

[0113] The feature extraction unit can be used to extract features from the reference image to obtain a feature map; and to extract features from the 3D point cloud data to obtain a point cloud feature map.

[0114] A positioning unit can be used to locate the target user based on the feature map and the point cloud feature map.

[0115] In some other embodiments, the positioning module may further include a weight map construction unit and a fusion unit.

[0116] The weight map construction unit can be used to copy the feature map N times by column to obtain an intermediate feature map, where N represents the total number of points in the point cloud feature map; and splice the intermediate feature map with the point cloud feature map to obtain a spliced ​​feature map; and normalize the spliced ​​feature map to obtain an attention weight map.

[0117] The fusion unit can be used to multiply the attention weight map with the corresponding elements of the point cloud feature map to obtain a fused feature map.

[0118] The positioning unit may also be configured to locate the target user based on the fused feature map.

[0119] In some other embodiments, the positioning module may further include a candidate frame determining unit and a target frame determining unit.

[0120] The candidate box determination unit can be used to generate a two-dimensional candidate box of the fused feature map using RPN; and calculate the three-dimensional candidate box corresponding to each two-dimensional candidate box in combination with the depth map; wherein the size of the three-dimensional candidate box is initialized based on the average size of each three-dimensional candidate box in the training set.

[0121] The target frame determination unit may be configured to adjust the three-dimensional candidate frame using a multi-task loss function to obtain a three-dimensional target frame of the target user.

[0122] The positioning unit may also be configured to position the target user based on the three-dimensional target frame.

[0123] It should be noted that the above-mentioned device may also include other implementations according to the description of the method embodiment. The specific implementation methods can refer to the description of the relevant method embodiment, and will not be described one by one here.

[0124] The service robot may also include at least one processor and a memory for storing processor executable instructions, and the instructions, when executed by the processor, implement the steps of the method described in any one or more of the above embodiments. The memory may include a physical device for storing information, which is usually to digitize the information and then store it in a medium using electrical, magnetic or optical methods. The storage medium may include: devices that store information using electrical energy, such as various memories, such as RAM, ROM, etc.; devices that store information using magnetic energy, such as hard disks, floppy disks, magnetic tapes, magnetic core memories, bubble memories, and U disks; devices that store information optically, such as CDs or DVDs. Of course, there are other types of readable storage media, such as quantum memories, graphene memories, and so on.

[0125] It should be noted that the embodiments of this specification are not limited to the situations that must comply with the standard data model / template or the embodiments of this specification. Certain industry standards or slightly modified implementation plans based on the implementation described in the custom method or the embodiment can also achieve the same, equivalent or similar, or predictable implementation effects after deformation of the above-mentioned embodiments. The embodiments obtained by using these modified or deformed data acquisition, storage, judgment, processing methods, etc. can still fall within the scope of the optional implementation plans of this specification.

[0126] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. In the description of this specification, the description of the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of this specification. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, in the absence of contradiction, a person skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0127] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A service resource allocation method applied to a service robot, characterized in that: The method comprises: Determine the location information of the target user to be processed according to the identification result of the user in the designated area, and move to the target user based on the location information; Determining the current business handling needs of the target user based on the communication information with the target user; Extracting at least one service device in the designated area that can meet the current business processing requirements as a candidate service device; Based on the processing progress of the existing business processing requirements of the candidate service devices, select service devices that provide services to the target user from the candidate service devices, so that the selected service devices provide services to the target user; Wherein, the service robot is equipped with a binocular camera, and the determining of the location information of the target user to be handled includes: Use binocular cameras to collect information about the user and obtain a left image and a right image; and use the left image or the right image as a reference image; Construct a depth map based on the left image and the right image, and perform feature extraction on the reference image to obtain a feature map; Copy the feature map N times by column to obtain an intermediate feature map, where N represents the total number of points in the point cloud feature map, and the intermediate feature map is an RGB feature map; Splicing the intermediate feature map with the point cloud feature map to obtain a spliced ​​feature map, wherein the point cloud feature map is obtained by extracting features from 3D point cloud data, and the 3D point cloud data is determined based on the reference map and the depth map; Normalizing the concatenated feature map to obtain an attention weight map; Multiplying the attention weight map with the corresponding elements of the point cloud feature map to obtain a fused feature map, wherein the attention weight map is used to guide the point cloud feature map to better express features; The target user is located based on the fused feature map.

2. The method according to claim 1, characterized in that The step of selecting a service device providing service to the target user from the candidate service devices includes: For any existing business processing demand of the candidate service device, obtain the business type to be processed of the existing business processing demand and the node to be processed in the business type being processed, so as to determine the remaining processing time of the existing business processing demand according to the business type to be processed and the processing time of the node to be processed; Based on the remaining processing time of the existing business handling demand of the candidate service device, a service device providing service to the target user is screened out from the candidate service devices.

3. The method according to claim 2, characterized in that Taking the service device that provides services to the target user as the target service device, the method further includes: Determining the waiting time of the current service processing demand based on the remaining processing time of the existing service processing demand of the target service device; The device identification information and waiting time of the target service device are fed back to the target user.

4. The method according to claim 1, characterized in that: The 3D point cloud data is determined according to the reference map and the depth map, including: In the camera coordinate system of the reference image, each pixel in the reference image and the depth image is back-projected into 3D coordinates to obtain 3D point cloud data.

5. The method according to claim 1, characterized in that The locating the target user based on the fused feature map includes: Generate a two-dimensional candidate box of the fused feature map using RPN; Calculating a three-dimensional candidate frame corresponding to each two-dimensional candidate frame in combination with the depth map; wherein the size of the three-dimensional candidate frame is initialized based on the average size of each three-dimensional candidate frame in the training set; The three-dimensional candidate frame is adjusted using a multi-task loss function to obtain a three-dimensional target frame of the target user; The target user is located according to the three-dimensional target frame.

6. A service robot, characterized in that: The robot comprises: A positioning module, used to determine the location information of a target user to be processed according to the identification result of the user in the specified area, so as to move to the target user based on the location information; A demand determination module, used to determine the current business handling demand of the target user based on the communication information with the target user; An extraction module, used to extract at least one service device in the specified area that can meet the current business processing requirements as a candidate service device; A screening module, for screening service devices providing services to the target user from the candidate service devices based on the processing progress of the existing business processing requirements of the candidate service devices, so that the screened service devices provide services to the target user; The service robot is equipped with a binocular camera, which is used to collect information about the user and obtain a left image and a right image; and the left image or the right image is used as a reference image; the positioning module includes: A point cloud image construction unit, configured to construct a depth map based on the left image and the right image; and determine 3D point cloud data based on the reference image and the depth map; A feature extraction unit, configured to extract features from the reference image to obtain a feature map; and to extract features from the 3D point cloud data to obtain a point cloud feature map; A positioning unit, used for positioning the target user based on the feature map and the point cloud feature map; Wherein, locating the target user based on the feature map and the point cloud feature map includes: Copy the feature map N times by column to obtain an intermediate feature map, where N represents the total number of points in the point cloud feature map, and the intermediate feature map is an RGB feature map; Splicing the intermediate feature map with the point cloud feature map to obtain a spliced ​​feature map; Normalizing the concatenated feature map to obtain an attention weight map; Multiplying the attention weight map with the corresponding elements of the point cloud feature map to obtain a fused feature map, wherein the attention weight map is used to guide the point cloud feature map to better express features; The target user is located based on the fused feature map.

7. The service robot according to claim 6, characterized in that: The step of determining the 3D point cloud data based on the reference image and the depth image includes: back-projecting each pixel point in the reference image and the depth image into 3D coordinates in a camera coordinate system of the reference image to obtain the 3D point cloud data.

8. The service robot according to claim 6, characterized in that: The positioning module also includes a candidate frame determination unit and a target frame determination unit; The candidate frame determination unit is used to generate a two-dimensional candidate frame of the fused feature map using RPN; and calculate a three-dimensional candidate frame corresponding to each two-dimensional candidate frame in combination with the depth map; wherein the size of the three-dimensional candidate frame is initialized based on the average size of each three-dimensional candidate frame in the training set; The target frame determination unit is used to adjust the three-dimensional candidate frame by using a multi-task loss function to obtain a three-dimensional target frame of the target user; The positioning unit is further used to position the target user according to the three-dimensional target frame.

Images