Multi-camera positioning and scheduling system and method
The multi-camera positioning and scheduling system generates a panoramic map and combines image recognition and planning algorithms to solve the positioning and scheduling problems of staff and self-driving vehicles in the warehousing space, improves the efficiency of warehousing management and saves human resources.
Patent Information
- Application Number
- CN202110005055.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2020-11-06
- Filing Date
- 2021-01-04
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-01-04
AI Technical Summary
The existing technology lacks effective methods to locate and dispatch staff and self-driving vehicles in the storage space, resulting in poor warehousing management efficiency and waste of human resources.
The multi-camera positioning and scheduling system is adopted to collect images of indoor space through the camera and generate panoramic maps. Combined with the deep learning algorithm for image recognition and the scheduling algorithm for warehousing resource planning and scheduling algorithm, the precise positioning and workload balance of work units are achieved, and the mobile route is optimized.
It improves the efficiency of warehousing management, saves human resources, realizes high-precision positioning and balanced workload of work units in indoor space, and optimizes the mobile path.
Smart Images

Figure CN114445502B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a multi-camera positioning and scheduling system and a method thereof. Background Art
[0002] E-commerce platforms have become one of the main shopping channels for modern people. Generally speaking, the number of goods that suppliers need to provide is extremely large; therefore, when orders suddenly increase, suppliers may have various storage problems (such as insufficient storage space or goods placed in the wrong location) that lead to poor storage management efficiency, affecting delivery time, and also consuming a lot of human resources.
[0003] Self-propelled vehicles can be used to perform highly repetitive tasks, thus saving a large amount of manpower. However, due to the current lack of methods for effectively locating and dispatching workers and self-propelled vehicles in storage spaces, the aforementioned problems cannot be effectively solved. Summary of the invention
[0004] An object of the present invention is to provide a multi-camera positioning and scheduling system and method thereof to generate a panoramic map of an indoor space and effectively locate units in the indoor space.
[0005] Another object of the present invention is to propose a multi-camera positioning and scheduling system and method thereof to effectively average the workload of all work units and optimize the movement routes to improve the efficiency of indoor management and save human resources.
[0006] According to one embodiment of the present invention, the present invention proposes a multi-camera positioning and scheduling system, which includes multiple cameras and a processing device. The cameras are distributed in an indoor space with multiple areas; the cameras correspond to the areas respectively and capture images of the areas respectively. The processing device converts the pixel coordinates of the image of the camera corresponding to each area into the camera coordinates of this area, and converts the camera coordinates of this area into the world coordinates of this area according to the relative position relationship between the camera and the indoor space, and superimposes and takes the intersection of the images of the cameras to integrate the images captured by the cameras, and generates a panoramic map of the indoor space defined by the world coordinate system. Wherein, when the image captured by any camera contains a working unit, the processing device projects the working unit to the panoramic map according to the pixel coordinates of the image captured by the camera.
[0007] According to another embodiment of the present invention, a multi-camera positioning and scheduling method is proposed, which includes the following steps: multiple cameras are set in an indoor space having multiple areas, and the cameras respectively correspond to the areas and respectively capture images of the areas; a processing device is used to convert the pixel coordinates of the image of the camera corresponding to each area into the camera coordinates of this area; the processing device is used to convert the camera coordinates of this area into the world coordinates of this area according to the relative position relationship between the camera and the indoor space; the processing device is used to superimpose and take the intersection of the images of the cameras to integrate the images captured by the cameras and generate a panoramic map of the indoor space defined by the world coordinate system; and when the image captured by any camera includes a working unit, the processing device projects the working unit onto the panoramic map according to the pixel coordinates of the working unit in the image captured by this camera.
[0008] The aforementioned indoor space includes but is not limited to the interior space of a warehouse, but refers to any indoor space, such as an office, a home, a factory, a store, a shopping mall, a department store, etc. The multi-camera positioning and scheduling system and method thereof can be applied to any indoor space.
[0009] As described above, the multi-camera positioning scheduling system and method according to the present invention may have one or more of the following advantages:
[0010] (1) In one embodiment of the present invention, the multi-camera positioning and scheduling system can superimpose and take the intersection of the images of these cameras, and integrate the images captured by all cameras to generate a panoramic map of the indoor space, and can project the work unit in the image of any camera onto the panoramic map, so that the work unit in the indoor space can be effectively positioned.
[0011] (2) In one embodiment of the present invention, the multi-camera positioning and scheduling system can use bounding boxes to mark the working units in the images of each camera when the images of two or more cameras include the working units, and project the bounding boxes onto the panoramic map, and project the working units onto the panoramic map according to the center points of the overlapping parts of the bounding boxes, thereby effectively increasing the positioning accuracy.
[0012] (3) In one embodiment of the present invention, the multi-camera positioning and scheduling system can analyze the order data, warehouse data and location of the work units of the warehouse management system to generate analysis results, and generate work allocation and mobile path planning based on the analysis results, so as to average the workload of the work units (staff and transportation vehicles, such as self-driving cars, etc.) and optimize the mobile routes, so as to improve the efficiency of indoor environment management and save human resources. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] Figure 1 It is a block diagram of a multi-camera positioning and scheduling system according to a first embodiment of the present invention.
[0014] Figure 2 It is a first schematic diagram of a multi-camera positioning and scheduling system according to a first embodiment of the present invention.
[0015] Figure 3A It is a first schematic diagram of generating a panoramic map of a multi-camera positioning and scheduling system according to the first embodiment of the present invention.
[0016] Figure 3B It is a second schematic diagram of generating a panoramic map of the multi-camera positioning and scheduling system according to the first embodiment of the present invention.
[0017] Figure 3C It is a third schematic diagram of generating a panoramic map of the multi-camera positioning and scheduling system according to the first embodiment of the present invention.
[0018] Figure 4A The flowchart is a monitoring method of a multi-camera positioning and scheduling system according to the first embodiment of the present invention.
[0019] Figure 4B Flow chart of the image recognition deep learning algorithm of the multi-camera positioning and scheduling system according to the first embodiment of the present invention.
[0020] Figure 5 It is a schematic diagram of an actual application scenario of the multi-camera positioning and scheduling system according to the first embodiment of the present invention.
[0021] Fig. 6A It is a schematic diagram of executing the warehouse resource planning and scheduling algorithm of the multi-camera positioning and scheduling system according to the first embodiment of the present invention.
[0022] Figure 6B It is a flowchart of the detailed steps of the warehouse resource planning and scheduling algorithm of the multi-camera positioning and scheduling system of the first embodiment of the present invention.
[0023] Fig. 7A It is a first schematic diagram of work allocation and movement path planning of a multi-camera positioning and scheduling system according to the first embodiment of the present invention.
[0024] Figure 7B It is a second schematic diagram of the work allocation and movement path planning of the multi-camera positioning and scheduling system according to the first embodiment of the present invention.
[0025] Figure 7C It is a third schematic diagram of the work allocation and movement path planning of the multi-camera positioning and scheduling system according to the first embodiment of the present invention.
[0026] Figure 8 Flow chart of the multi-camera positioning scheduling method according to the first embodiment of the present invention.
[0027] Fig. 9 FIG. 4 is a block diagram of a multi-camera positioning and scheduling system according to a second embodiment of the present invention.
[0028] Fig. 10A It is a first schematic diagram of generating a panoramic map of a multi-camera positioning and scheduling system according to the second embodiment of the present invention.
[0029] Fig. 10B It is a second schematic diagram of generating a panoramic map of the multi-camera positioning and scheduling system according to the second embodiment of the present invention.
[0030] Fig.11A It is a first schematic diagram of a positioning algorithm of a multi-camera positioning and scheduling system according to a second embodiment of the present invention.
[0031] Fig. 11B It is a second schematic diagram of the positioning algorithm of the multi-camera positioning and scheduling system according to the second embodiment of the present invention.
[0032] Fig.12 Flow chart of a multi-camera positioning scheduling method according to a second embodiment of the present invention.
[0033] Description of Reference Numerals
[0034] 1, 2 Multi-camera positioning and scheduling system
[0035] 11,21 Camera
[0036] 12, 22 Processing device
[0037] 13, 23 Computing device
[0038] M Warehouse Management System
[0039] S Interior Space
[0040] O origin
[0041] xx axis
[0042] yy-axis
[0043] ZZ axis
[0044] U Work Unit
[0045] G1~G3 Work unit group
[0046] L Shelves
[0047] Map Panoramic map
[0048] R1, R1', R2, R3, Rs1, Rs2, Rs3 path
[0049] Pt Products
[0050] K Tally Area
[0051] F Bounding Box
[0052] J Overlapping part
[0053] Md Center Point
[0054] Da1 Order Information
[0055] Da2 Warehouse Information
[0056] Db, Rb Coordinates of working units
[0057] SA Warehouse Resource Planning and Scheduling Algorithm
[0058] Pr work allocation and mobile path planning
[0059] T1~T3 Considerations for mobile path planning
[0060] A. Robotic Arm DETAILED DESCRIPTION
[0061] The following will refer to the relevant drawings to illustrate the embodiments of the multi-camera positioning and scheduling system and method according to the present invention. For the sake of clarity and convenience of illustration, the components in the drawings may be exaggerated or reduced in size and proportion. In the following description and / or patent application, when an element is referred to as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element or there may be an intervening element; and when an element is referred to as being "directly connected" or "directly coupled" to another element, there is no intervening element, and other words used to describe the relationship between elements or layers should be interpreted in the same way. For ease of understanding, the same elements in the following embodiments are illustrated with the same symbols.
[0062] See also Figure 1 and Figure 2 , which is a block diagram and schematic diagram of a multi-camera positioning and scheduling system of a first embodiment of the present invention. As shown in the figure, the multi-camera positioning and scheduling system 1 includes a plurality of cameras 11, a processing device 12 and a computing device 13, and is connected to a warehouse management system M. The cameras 11 are connected to the processing device 12. The processing device 12 is connected to the computing device 13. The computing device 13 is connected to the warehouse management system M.
[0063] The cameras 11 are distributed in the indoor space S. The indoor space S can be divided into a plurality of areas, and the cameras 11 correspond to the areas respectively; wherein the areas are defined as the areas covered by the images of each camera 11. Since the cameras 11 correspond to the areas of the indoor space S respectively, the world coordinates of each camera 11 and the relative position relationship between each camera 11 and the indoor space S are known. The aforementioned indoor space S is not limited to the interior space of a warehouse, but refers to any indoor space, such as an office, a home, a factory, a store, a shopping mall, and a department store. The multi-camera positioning and scheduling system 1 can be applied to any indoor space.
[0064] The specific point in the indoor space S is the origin O, and the x-axis, y-axis and z-axis are defined based on the origin O to establish a world coordinate system, thereby representing the coordinates of each position in the indoor space S in the real space. If applied to the pinhole camera model, the processing device 12 can perform the corresponding coordinate transformation, as shown in the following formula (1):
[0065]
[0066] Wherein, K represents the internal parameter matrix of the camera 11; α x and α y represents the focal length of the lens of the camera 11; u0 and v0 represent the coordinates of the center point of the image of the camera 11.
[0067] Therefore, the processing device 12 can convert the pixel coordinates of the corresponding area into the camera coordinates of the area according to the internal parameter matrix of each camera 11, and then convert the camera coordinates of the area into the world coordinates of the area in the image of the camera 11, as shown in the following formula (2):
[0068]
[0069] Among them, (X c , Y c , Z c ) is the camera coordinate of any position in this area, (X w , Y w , Z w ) represents the world coordinates corresponding to this position; R and T are the external parameters of this camera 11; wherein, R represents the rotation parameter of this camera 11; T represents the translation parameter of this camera 11.
[0070] The complete calculation process is shown in formula (3):
[0071]
[0072] Formula (3) defines the 3×4 projection matrix from Euclidean 3D space to an image, as shown in Formula (4):
[0073]
[0074] Wherein, x represents the pixel coordinates of each position in this area, which can be expressed as (u, v), that is, the 1×3 matrix in formula (3); X(=(X w , Y w , Z w )) is the world coordinate corresponding to this position; as mentioned above, K represents the internal parameter matrix of the camera 11; R and T are the external parameters of this camera 11.
[0075] If applied to a fisheye camera model, for example, the processing device 12 may also perform a corresponding coordinate transformation. c = 1 has a given point, the world coordinates of this given point are X = (X w , Y w , Z w ), its pinhole projection is shown in the following formula (5):
[0076]
[0077] Formula (5) can be further expressed as the following formula (6):
[0078] f=r′·tan(θ)................................................(6)
[0079] in, θ represents the angle of each position in this area relative to the lens; r represents the distance of this position relative to the center of the lens; f represents the scaling parameter; (u, v) represents the pixel coordinates of each position in this area.
[0080] Therefore, the fisheye distortion can be expressed by the following equation (7):
[0081] θ d =θ(1+k1·θ 2 +k2·θ 4 +k3·θ 6 +k4·θ 8 )..................(7)
[0082] Among them, θ d Indicates fisheye distortion; k1~k4 represent the structural parameters of the lens.
[0083] Therefore, the coordinates of the twist point can be expressed by the following formula (8):
[0084]
[0085] Among them, (x d ,y d ) represents the coordinates of the distortion point.
[0086] Convert equation (8) to pixel coordinates; the final pixel coordinate vector can be expressed by the following equation (9):
[0087]
[0088] (u, v) represents the pixel coordinates of each position in this area; f x represents the scaling parameter; α represents the deformation parameter; c x and c y Represents the displacement parameter.
[0089] Formula (9) can be expressed in matrix form by the following formula (10):
[0090]
[0091] Therefore, the processing device 12 can convert the pixel coordinates of any area into the camera coordinates of this area, and then convert the camera coordinates of this area into the world coordinates of this area in the image of this camera 11.
[0092] Next, the processing device 12 can superimpose and take the intersection of the images of the cameras 11 to integrate the images captured by the cameras 11 to generate a panoramic map of the indoor space S defined by the world coordinate system. In one embodiment, the processing device 12 can be various computer devices, such as a server, a notebook computer, or a personal computer. Among them, the processing device 12 can obtain the relative position relationship between the camera 11 and the indoor space S according to the feature points in the image of each camera 11; then, the processing device 12 can generate a panoramic map according to the relative position relationship between the cameras 11 and the indoor space S by taking the intersection of the images of the cameras 11 and superimposing the images of the cameras 11.
[0093] When the image captured by any camera 11 includes a work unit U, the processing device 12 projects the work unit U onto the panoramic map according to the pixel coordinates of the image captured by the camera 11 and the relative position relationship between the camera and the indoor space S, and simultaneously obtains the world coordinates of the work unit U; that is, the panoramic map coordinate system of the panoramic map is the world coordinate system, and the panoramic map coordinates of the panoramic map coordinate system are the world coordinates. Therefore, the multi-camera positioning and scheduling system 1 can locate all the work units U located in the indoor space S. In one embodiment, the work unit U can be, but is not limited to, a staff member or a transport vehicle (such as a self-driving car, etc.); the work unit U can be any object in the indoor space S.
[0094] In the above steps, the processing device 12 can execute an image recognition deep learning algorithm to mark the work unit U in the image of each camera 11 with a bounding box, and process the image of each camera 11 through a convolutional neural network (CNN) to calculate the probability that the work unit U in the image of each camera 11 is a staff member and the probability that the work unit U is a transport vehicle, so as to determine whether the work unit U is a staff member or a transport vehicle, and output the judgment result and its coordinates.
[0095] See also Figure 3A , Figure 3B and Figure 3C , which are the first schematic diagram, the second schematic diagram and the third schematic diagram of generating a panoramic map of a multi-camera positioning and scheduling system according to the first embodiment of the present invention. Figure 3A As shown, the cameras 11 distributed in the indoor space S correspond to different areas and capture images of these areas respectively; wherein the areas are the areas covered by the images of each camera 11 .
[0096] like Figure 3B As shown, the processing device 12 converts the pixel coordinates of each position of the image of the camera 11 corresponding to each area into the camera coordinates of each position in the lens of the camera 11 according to formula (1), and then converts the camera coordinates of each position in the area into the world coordinates of each position in the area according to formula (2). Then, the processing device 12 generates the following by taking the intersection of the images of the cameras 11 and superimposing the images of the cameras 11. Figure 3C The above procedure includes the following steps:
[0097] Step S31: multiple cameras are set in an indoor space having multiple areas, and the cameras correspond to the areas respectively and capture images of the areas respectively.
[0098] Step S32: using a processing device to convert the pixel coordinates of each position of the image of the camera corresponding to each area into the camera coordinates of this area.
[0099] Step S33: The processing device converts the camera coordinates of this area into the world coordinates of this area according to the relative position relationship between the camera and the indoor space.
[0100] Step S34: The images of the cameras are superimposed and the intersections are taken by the processing device to integrate the images of the cameras and generate a panoramic map of the indoor space defined by the world coordinate system.
[0101] Step S35: When the image captured by any camera includes a working unit, the processing device projects the working unit onto the panoramic map according to the pixel coordinates of the image captured by the camera.
[0102] like Figure 3C As shown, when there are working units U in the images of the cameras 11, the processing device 12 can project the working units U to corresponding positions of the panoramic map Map according to the pixel coordinates of the working units U in the cameras 11 and the relative relationship between the cameras 11 and the panoramic map, so as to locate the working units U.
[0103] Through the above mechanism, the multi-camera positioning and scheduling system 1 can generate a panoramic map of the indoor space S through the cameras 11 and obtain the real coordinates of the work units U and the shelves L adjacent thereto.
[0104] See also Figure 4A and Figure 4B , which is a flow chart of the monitoring method of the multi-camera positioning and scheduling system and a flow chart of the image recognition deep learning algorithm of the first embodiment of the present invention. As mentioned above, the multi-camera positioning and scheduling system 1 of this embodiment adopts a special monitoring method; the multi-camera positioning and scheduling system 1 can receive images from multiple cameras 11 set in the indoor space S, and then convert the images of these cameras 11 into a panoramic map of the indoor space S defined by the world coordinate system, and project these work units U onto the panoramic map, and at the same time obtain the world coordinates of these work units U to locate these work units U.
[0105] In addition, as mentioned above, the multi-camera positioning and scheduling system 1 recognizes the work units U in the images of the cameras 11 as workers or transport vehicles through image recognition deep learning algorithms. Figure 4A and Figure 4B The monitoring method and the image recognition deep learning algorithm of the multi-camera positioning and scheduling system 1 of this embodiment are respectively described.
[0106] like Figure 4AAs shown, the monitoring method adopted by the multi-camera positioning and scheduling system 1 of this embodiment includes the following steps:
[0107] Step S41: multiple cameras are set in an indoor space having multiple areas, and the cameras correspond to the areas respectively and capture images of the areas respectively.
[0108] Step S42: Convert the images of the cameras into a panoramic map by a processing device.
[0109] Step S43: Execute an image recognition deep learning algorithm via a processing device and recognize the working units in the images of each camera through a convolutional neural network.
[0110] Step S44: Outputting the panoramic map coordinates (ie, world coordinates) of the working units in the images of each camera through the processing device.
[0111] like Figure 4B As shown, the aforementioned image recognition deep learning algorithm includes the following steps:
[0112] Step S421: Input the images from the cameras into a convolutional neural network via a processing device.
[0113] Step S422: The processing device calculates the probability that the work unit in the image of each camera marked by the bounding box is a worker and the probability that the work unit is a transport vehicle to identify the work unit.
[0114] Step S423: Generate an identification result through the processing device.
[0115] Step S424: The processing device generates coordinates of the working units of the images of the cameras.
[0116] Next, if Figure 1 As shown, the computing device 13 analyzes the order data, warehouse data (including product location information) of the warehouse management system M and the locations of the work units U in the panoramic map Map to generate analysis results. The computing device 13 can analyze the above data through a deep reinforcement learning algorithm to generate analysis results; in one embodiment, the deep reinforcement learning algorithm can be a Deep Q-Learning algorithm or other similar algorithms. In one embodiment, the computing device 13 can be various computer devices, such as a server, a notebook computer, or a personal computer.
[0117] Then, the computing device 13 can execute the warehouse resource planning and scheduling algorithm to generate work allocation and mobile path planning according to the analysis results. Among them, the computing device 13 can execute the pointer network algorithm according to the analysis results to generate work allocation and mobile path planning; the computing device 13 can also execute other existing algorithms, such as the algorithm disclosed in Taiwan Patent Application No. 107130814.
[0118] See also Figure 5 , which is a schematic diagram of an actual application scenario of the multi-camera positioning and scheduling system of the first embodiment of the present invention. Figure 5 An example is given of a practical application scenario in which the multi-camera positioning and scheduling system 1 of this embodiment executes the aforementioned monitoring method and the warehouse resource planning and scheduling algorithm in a logistics warehouse.
[0119] As shown in the figure, the warehouse management system M outputs the order data Da1 and the warehouse data Da2, and the images of the cameras 11 are obtained through the aforementioned monitoring method (such as Figure 2 , Figure 3A-3B and Figures 4A to 4B Then, the computing device 13 can execute the warehouse resource planning and scheduling algorithm SA (which will be Figures 6A-6B Detailed description) generates work allocation and mobile path planning Pr; the purpose of work allocation and mobile path planning Pr is to evenly distribute the workload of the work units U, optimize the mobile paths of the work units U, and effectively avoid the collision of the work units U and the situation of meeting when the control routes overlap. Next, the work units U in the logistics warehouse perform work according to the aforementioned work allocation and mobile path planning Pr, and the cameras 11 continuously monitor the work units U in real time and update the coordinates Db of the work units, so that the computing device 13 can execute the warehouse resource planning and scheduling algorithm SA according to the order data Da1, the warehouse data Da2 and the updated coordinates Rb of the work units to continuously update the work allocation and mobile path planning Pr.
[0120] See also Fig. 6A and Figure 6B , which is a schematic diagram of executing the warehouse resource planning and scheduling algorithm of the multi-camera positioning and scheduling system of the first embodiment of the present invention and a flowchart of the detailed steps of the warehouse resource planning and scheduling algorithm. Fig. 6AAs shown, the computing device 13 can receive the order data Da1, the warehouse data Da2 and the coordinates Db of the work units to execute the warehouse resource planning and scheduling algorithm SA, and can execute the warehouse resource planning and scheduling algorithm SA according to the order data Da1, the warehouse data Da2 and the coordinates Db of the work units and generate work allocation and mobile path planning Pr.
[0121] like Figure 6B As shown, the detailed steps of the warehouse resource planning and scheduling algorithm are as follows:
[0122] Step S61: Input order information, warehouse information and the coordinates of the work units.
[0123] Step S62: Analyze the order data, warehouse data and the coordinates of the work units.
[0124] Step S63: Generate work allocation and mobile path planning. The mobile path planning considerations may include one or more of the shortest path T1, collision prevention and oncoming vehicle control T2, and the load capacity of the transport vehicle T3.
[0125] Step S64: Outputting the work allocation and movement path planning to the work units.
[0126] See also Fig. 7A , Figure 7B and Figure 7C , which are the first schematic diagram, the second schematic diagram and the third schematic diagram of the work allocation and mobile path planning of the multi-camera positioning and scheduling system of the first embodiment of the present invention. Fig. 7A As shown, nine products Pt on the order are distributed on different shelves L. A plurality of work unit groups G1, G2, G3 need to reach the locations of the products Pt via paths R1, R2, and R3 respectively, and the robot arms A transport the products Pt.
[0127] However, if proper work allocation and movement path planning are not performed, the work unit groups G1, G2, and G3 cannot complete the work efficiently. Figure 7B As shown, the work unit group G1 needs to first transport three commodities Pt and move to the tally area K via the path R1, and then transport two commodities Pt and move to the tally area K via the path R1'; while the work unit group G2 and the work unit group G3 transport two commodities Pt and move to the tally area K via the paths R2 and R3 respectively. However, the workload of the work unit group G1 is obviously greater than that of the work unit groups G2 and G3, so the work distribution of the work unit groups G1, G2, and G3 is not even, and the work cannot be completed via the shortest path.
[0128] like Figure 7CAs shown, the computing device 13 can execute the warehouse resource planning and scheduling algorithm to generate work allocation and mobile path planning according to the analysis results. Among them, the computing device 13 can execute the pointer network algorithm according to the analysis results to generate work allocation and mobile path planning; the computing device 13 can also execute other existing algorithms, such as the algorithm disclosed in Taiwan Patent Application No. 107130814. Through the above mechanism, the computing device 13 can average the workload of the work unit groups G1, G2, and G3, so that the three work unit groups G1, G2, and G3 can transport three commodities Pt respectively, and move to the tally area K respectively along the shortest paths Rs1, Rs2, and Rs3. Therefore, the multi-camera positioning and scheduling system 1 can effectively improve the efficiency of indoor environment management and save human resources.
[0129] Since there is currently a lack of methods for effectively locating and dispatching staff and self-driving vehicles in indoor spaces, suppliers may have poor warehouse management efficiency due to various warehouse problems, which affects the delivery time and also consumes a lot of human resources. However, according to an embodiment of the present invention, the multi-camera positioning and scheduling system 1 can superimpose and take the intersection of the images of the cameras 11 to integrate the images captured by all cameras 11 to generate a panoramic map Map of the indoor space S, and can project the work unit U in the image of any camera 11 to the panoramic map Map, and can effectively locate the work unit U in the indoor space S. In addition, the multi-camera positioning and scheduling system 1 can analyze the order data Da1, warehouse data Da2 and coordinates Db of the work unit U of the warehouse management system M to generate analysis results, and generate work allocation and mobile path planning Pr based on the analysis results, so that the workload of the work units U (the work unit U can be but not limited to staff or transportation vehicles, such as self-driving vehicles, etc.) can be averaged and the mobile routes can be optimized to improve the efficiency of indoor environment management and save human resources.
[0130] This embodiment is for illustration only. The various components of the multi-camera positioning and scheduling system 1 and their coordination relationships may vary according to actual needs, and the present invention is not limited thereto.
[0131] See also Figure 8 , which is a flow chart of a multi-camera positioning and scheduling method of a first embodiment of the present invention. The multi-camera positioning and scheduling method of the multi-camera positioning and scheduling system 1 of this embodiment comprises the following steps:
[0132] Step S81: multiple cameras are set in an indoor space having multiple areas, and the cameras correspond to the areas respectively and capture images of the areas respectively.
[0133] Step S82: using a processing device to convert the pixel coordinates of the image of the camera corresponding to each area into the camera coordinates of this area.
[0134] Step S83: The processing device converts the camera coordinates of this area into the world coordinates of this area according to the relative position relationship between this camera and the indoor space.
[0135] Step S84: The processing device is used to superimpose and take the intersection of the images of the cameras to integrate the images captured by the cameras and generate a panoramic map of the indoor space defined by the world coordinate system.
[0136] Step S85: When the image captured by any camera includes a working unit, the processing device projects the working unit onto the panoramic map according to the pixel coordinates of the image captured by the camera.
[0137] Step S86: Analyze the order data, warehouse data and the positions of the work units in the panoramic map of the warehouse management system through the computing device to generate analysis results.
[0138] Step S87: Generate work allocation and movement path planning based on the analysis results through the computing device.
[0139] See also Fig. 9 , Fig. 10A and Fig. 10B ; Fig. 9 is a block diagram of a multi-camera positioning and scheduling system according to a second embodiment of the present invention; Fig. 10A and Fig. 10B The first schematic diagram and the second schematic diagram of the multi-camera positioning and scheduling system for generating a panoramic map of the second embodiment of the present invention are shown in FIG6 . As shown in FIG6 , the multi-camera positioning and scheduling system 2 includes a plurality of cameras 21, a processing device 22, and a computing device 23. The cameras 21 are connected to the processing device 22. The processing device 22 is connected to the computing device 23. The computing device 23 is connected to the warehouse management system M.
[0140] like Fig. 10AAs shown, the cameras 21 are distributed in the indoor space S. The indoor space S can be divided into multiple areas, and the cameras 21 correspond to the areas and capture images of the areas respectively; wherein the areas are the areas covered by the images of each camera 11. The images captured by any camera 21 may partially overlap with the images captured by at least one other camera 21. Similarly, the processing device 22 of the multi-camera positioning and scheduling system 2 can convert the pixel coordinates of each position of the image of the camera 21 corresponding to each area into the camera coordinates of each position of the area in the lens of the camera 21 according to formula (1), and then convert the camera coordinates of each position of the area into the world coordinates of each position of the area according to formula (2). Then, the processing device 22 generates a panoramic map by taking the intersection of the images of the cameras 21 and superimposing the images of the cameras 21. The aforementioned indoor space S is not limited to the interior space of a warehouse, but refers to any indoor space, such as an office, a home, a factory, a store, a shopping center, and a department store. The multi-camera positioning and scheduling system 2 can be applied to any indoor space.
[0141] Then, if Fig. 10B As shown, when the image captured by any camera 21 includes a work unit U, the processing device 22 of the multi-camera positioning and scheduling system 2 projects the work unit U onto the panoramic map Map according to the pixel coordinates of the image captured by the camera 21 and the relative position relationship between the camera 21 and the indoor space S, and simultaneously obtains the world coordinates of the work unit U; therefore, the multi-camera positioning and scheduling system 2 can locate all work units U located in the indoor space S.
[0142] The operation of the above-mentioned components is similar to that of the above-mentioned embodiment, so it will not be described in detail here. Different from the first embodiment, this embodiment adopts a more accurate image recognition algorithm.
[0143] See also Fig.11A and Fig. 11B , which are the first schematic diagram and the second schematic diagram of the positioning algorithm of the multi-camera positioning and scheduling system of the second embodiment of the present invention. Fig.11A As shown, the work unit U moves to a certain point in the indoor space S and is located within the shooting range of two cameras 21 set in the indoor space S, so that the images of the cameras 21 all include the work unit U.
[0144] like Fig. 11BAs shown, the processing device 22 marks the working unit U in the image of each camera 21 with the bounding box F, and projects the two bounding boxes F to the panoramic map Map at the same time. Then, the processing device 22 projects the working unit U to the panoramic map Map according to the center point Md of the overlapping part J of the two bounding boxes F. Among them, the processing device 22 can determine whether the working units U of the bounding boxes F are the same working unit U according to the position and moving direction of the working units U in the bounding boxes F.
[0145] Similarly, when the images of three or more cameras 21 include the same work unit U, the processing device 21 uses a bounding box F to mark the work unit U in the image of each camera 21, and simultaneously projects the three bounding boxes F to the panoramic map Map, and projects the work unit U to the panoramic map Map according to the center point of the overlapping part J of the three bounding boxes F.
[0146] Through the above-mentioned positioning algorithm, the multi-camera positioning and scheduling system 2 can more accurately locate all work units U in the indoor space S, and more accurately perform work allocation and mobile path planning to improve the efficiency of indoor environment management and save human resources.
[0147] This embodiment is for illustration only. Each component of the multi-camera positioning and scheduling system 2 and their coordination relationship may be changed according to actual needs, and the present invention is not limited thereto.
[0148] See also Fig.12 , which is a flow chart of a multi-camera positioning and scheduling method of a second embodiment of the present invention. The multi-camera positioning and scheduling method of the multi-camera positioning and scheduling system 2 of this embodiment comprises the following steps:
[0149] Step S121: multiple cameras are set in an indoor space having multiple areas, and the cameras correspond to the areas respectively and capture images of the areas respectively.
[0150] Step S122: using a processing device to convert the pixel coordinates of the image of the camera corresponding to each area into the camera coordinates of the area.
[0151] Step S123: The processing device converts the camera coordinates of the area into the world coordinates of the area according to the relative position relationship between the camera and the indoor space.
[0152] Step S124: The images captured by the cameras are integrated by superimposing and intersecting the images of the cameras through a processing device, and a panoramic map of the indoor space defined by a world coordinate system is generated.
[0153] Step S125: When the images from two or more cameras include a working unit, the processing device uses a bounding box to mark the working unit in the images from each camera.
[0154] Step S126: Projecting the bounding boxes onto the panoramic map by the processing device, and projecting the work unit onto the panoramic map according to the center point of the overlapping parts of the bounding boxes.
[0155] Step S127: Analyze the order data, warehouse data and the positions of the work units in the panoramic map of the warehouse management system through the computing device to generate analysis results.
[0156] Step S128: Generate work allocation and movement path planning according to the analysis results through the computing device.
[0157] In summary, according to an embodiment of the present invention, the multi-camera positioning and scheduling system can superimpose and take the intersection of the images of those cameras to integrate the images captured by all cameras to generate a panoramic map of the indoor space, and can project the work units in the image of any camera onto the panoramic map, so that the work units in the indoor space can be effectively positioned.
[0158] In addition, according to an embodiment of the present invention, the multi-camera positioning and scheduling system can use bounding boxes to mark the working units in the images of each camera when the images of two or more cameras include the working units, and project the bounding boxes onto the panoramic map, and project the working units onto the panoramic map according to the center points of the overlapping parts of the bounding boxes, thereby effectively increasing the positioning accuracy.
[0159] In addition, according to an embodiment of the present invention, the multi-camera positioning and scheduling system can analyze the order data, warehouse data and location of the work units of the warehouse management system to generate analysis results, and generate work allocation and mobile path planning based on the analysis results, so as to average the workload of those work units (work units can be but are not limited to staff and transportation vehicles, such as self-driving cars, etc.) and optimize the movement routes, so as to improve the efficiency of indoor environment management and save human resources.
[0160] It can be seen that the present invention has indeed achieved the desired improved technical effect by breaking through the existing technology, and it is not easy for technical personnel in this field to think of it. Its progressiveness and practicality obviously meet the application requirements for a patent, so a patent application is filed in accordance with the law.
[0161] The above description is for illustrative purposes only and is not intended to be limiting. Any other equivalent modifications or changes made thereto without departing from the spirit and scope of the present invention should be included in the scope of the appended patent applications.
Claims
1. A multi-camera positioning and scheduling system, characterized in that: include: A plurality of cameras are distributed in an indoor space having a plurality of areas, the cameras respectively correspond to the areas and respectively capture images of the areas, and an image captured by any one of the cameras partially overlaps with an image captured by another one of the cameras; and A processing device converts pixel coordinates of the image captured by the camera corresponding to each area into camera coordinates of the area, and converts the camera coordinates of the area into world coordinates of the area according to the relative position relationship between the camera and the indoor space, and takes the intersection of the images of the cameras and superimposes the images of the cameras to generate a panoramic map of the indoor space defined by the world coordinate system; Among them, when the image captured by any of the cameras includes a work unit, the processing device projects the work unit to the panoramic map according to the pixel coordinates of the work unit in the image captured by the camera; when the images captured by two or more of the cameras include the work unit, the processing device marks the work unit in each of the images captured by the cameras with a bounding box, projects the bounding boxes to the panoramic map, and projects the work unit to the panoramic map according to the center point of the overlapping part of the bounding boxes.
2. The multi-camera positioning and scheduling system according to claim 1, characterized in that: The processing device obtains the relative position relationship between the camera and the indoor space according to the feature points in the images captured by each camera.
3. The multi-camera positioning and scheduling system according to claim 1, characterized in that: The processing device converts the world coordinates of the area corresponding to the camera into the camera coordinates of the area according to the rotation parameters and translation parameters of each camera.
4. The multi-camera positioning and scheduling system according to claim 1, characterized in that: The processing device converts the pixel coordinates of the corresponding area into the camera coordinates of the area according to the internal parameter matrix of each camera.
5. The multi-camera positioning and scheduling system according to claim 1, characterized in that: It also includes a computing device, which analyzes the order information, storage information and the position of the work unit in the panoramic map of the warehouse management system to generate an analysis result, and generates work allocation and mobile path planning based on the analysis result.
6. The multi-camera positioning and scheduling system according to claim 5, characterized in that: The computing device executes a deep reinforcement learning algorithm to analyze the order data, the warehouse data, and the world coordinates of the work unit to generate the analysis result.
7. The multi-camera positioning and scheduling system according to claim 5, characterized in that: The computing device executes a pointer network algorithm according to the analysis result to generate the work allocation and the moving path planning.
8. The multi-camera positioning and scheduling system according to claim 1, characterized in that: The processing device executes an image recognition deep learning algorithm and identifies the work unit through a convolutional neural network.
9. A multi-camera positioning and scheduling method, characterized in that: include: A plurality of cameras are arranged in an indoor space having a plurality of areas, wherein the cameras correspond to the areas respectively and capture images of the areas respectively, and an image captured by any one of the cameras partially overlaps with an image captured by another one of the cameras; Using a processing device, the pixel coordinates of the image captured by the camera corresponding to each of the regions are converted into the camera coordinates of the region; The processing device converts the camera coordinates of the area into the world coordinates of the area according to the relative position relationship between the camera and the indoor space; The processing device takes the intersection of the images of the cameras and superimposes the images of the cameras to generate a panoramic map of the indoor space defined by a world coordinate system; as well as When any image captured by the camera includes a working unit, the processing device projects the working unit onto the panoramic map according to the pixel coordinates of the working unit in the image captured by the camera; The method further includes: When the images from two or more cameras include the working unit, the processing device marks the working unit in the images from each of the cameras with a bounding box; and The processing device projects the bounding boxes onto the panoramic map, and projects the working unit onto the panoramic map according to the center point of the overlapping parts of the bounding boxes.
10. The multi-camera positioning and scheduling method according to claim 9, characterized in that: The step of converting the camera coordinates of the camera into the pixel coordinates of the area according to the relative position relationship between the camera and the indoor space by the processing device further comprises: The processing device obtains the relative position relationship between the camera and the indoor space according to the feature points in the images captured by each camera.
11. The multi-camera positioning and scheduling method according to claim 9, characterized in that: The step of converting the world coordinates of the area corresponding to each camera into the camera coordinates of the area by the processing device also includes: The processing device converts the world coordinates of the area corresponding to the camera into the camera coordinates of the area according to the rotation parameters and translation parameters of each camera.
12. The multi-camera positioning and scheduling method according to claim 9, characterized in that: The step of converting the camera coordinates of the area into pixel coordinates of the area by the processing device also includes: The processing device converts the pixel coordinates of the corresponding area into the camera coordinates of the area according to the internal parameter matrix of each camera.
13. The multi-camera positioning and scheduling method according to claim 9, characterized in that: Also includes: The computing device analyzes the order data, the warehouse data and the location of the work unit on the panoramic map of the warehouse management system to generate an analysis result; as well as The computing device generates work allocation and movement path planning according to the analysis result.
14. The multi-camera positioning and scheduling method according to claim 13, characterized in that: The step of analyzing the order data of the warehouse management system, the warehouse data and the location of the work unit on the panoramic map by the computing device to generate the analysis result also includes: The computing device executes a deep reinforcement learning algorithm to analyze the order data, the warehouse data, and the world coordinates of the work unit to generate the analysis result.
15. The multi-camera positioning and scheduling method according to claim 13, characterized in that: The step of generating the work allocation and the moving path planning according to the analysis result by the computing device also includes: The computing device executes a pointer network algorithm according to the analysis result to generate the work allocation and the moving path planning.
16. The multi-camera positioning and scheduling method according to claim 9, characterized in that: Also includes: The processing device executes an image recognition deep learning algorithm and identifies the work unit through a convolutional neural network.
