Method and device for multi-modal indoor positioning of personnel in clean rooms

Abstract

The present invention provides a method and device for multimodal indoor positioning of personnel in a clean room, wherein multiple UWB base stations and multiple image acquisition devices are arranged in the clean room, the image acquisition device is arranged on the top of the clean room, the personnel in the clean room wear UWB tags, and the transmission power of the UWB base station and the UWB tag is lower than the first transmission power, including: the image acquisition device acquires image information in a target area at a first frequency to obtain a first positioning of the personnel in the target area in a first time period; the UWB base station acquires a first signal of the UWB tag in the target area at a second frequency, and obtains a second positioning and identity information of the personnel in the target area in the first time period according to the first signal; and the final positioning of the personnel in the target area is determined by combining the first positioning, the second positioning and the identity information. The present invention accurately locates personnel in a clean room containing a large amount of metal environment by combining the image acquisition device and the adjusted UWB base station.

Description

Technical Field

[0001] The present invention relates to the field of laboratory management, and in particular to a method and a device for multi-modal indoor positioning of personnel in a clean room. Background Art

[0002] At present, indoor personnel positioning technology mainly adopts video personnel positioning or wireless personnel positioning technology (for example, ultra-wideband positioning technology, Wifi positioning technology, RFID positioning technology, etc.).

[0003] Among them, video personnel positioning uses cameras or monitoring equipment to monitor, track and locate personnel in real time, and uses computer vision and image processing algorithms to analyze personnel images in the video stream to identify and track the position and movement trajectory of personnel in the video monitoring area. Video personnel positioning has the advantages of visualization, high real-time performance, and high precision; however, the disadvantage of video personnel positioning is that it is difficult to achieve continuous tracking of personnel across cameras in a large range, especially trajectory tracking and identity recognition in multi-person scenarios.

[0004] Wireless positioning technology determines the target location based on the characteristics of receiving and analyzing wireless signals, and achieves positioning through different algorithms. Taking ultra-wideband positioning technology as an example, ultra-wideband wireless signals (usually radio frequencies greater than 500 MHz) are used to achieve positioning. Time difference measurement or pulse response positioning methods are usually used to determine the location of the target by measuring the arrival time, amplitude, phase and other information of the signal. Ultra-wideband positioning technology has the advantages of large positioning range and no obstruction in general scenarios due to its strong signal propagation and penetration capabilities. However, in special scenarios with a large amount of metal, such as clean rooms, most of the floors, ceilings, walls and indoor equipment are made of metal, which will cause severe absorption, reflection, refraction and scattering of wireless signals, ultimately leading to multipath effects, which seriously reduce positioning accuracy and reliability. Summary of the invention

[0005] Based on this, the present invention provides a method and device for multi-modal indoor positioning of personnel in a clean room, which accurately positions personnel in a clean room containing a large amount of metal environment by combining an image acquisition device and an adjusted UWB base station.

[0006] In a first aspect, the present invention provides a method for multimodal indoor positioning of personnel in a clean room, wherein a plurality of UWB base stations and a plurality of image acquisition devices are arranged in the clean room, the image acquisition device is arranged on the top of the clean room, the personnel in the clean room wear a UWB tag, and the transmission power of the UWB base station and the UWB tag is lower than a first transmission power, and the method comprises:

[0007] The image acquisition device acquires image information in the target area at a first frequency to obtain a first location of a person in the target area within a first time period;

[0008] The UWB base station collects a first signal of a UWB tag in a target area at a second frequency, and obtains second positioning and identity information of a person in the target area in a first time period according to the first signal;

[0009] The final location of the person in the target area is determined by combining the first location, the second location and the identity information.

[0010] Furthermore, the determining of the final location of the person in the target area by combining the first location, the second location and the identity information is specifically as follows:

[0011] If the distance between the person in the target area and the image acquisition device is less than a first threshold, determining whether the distance between the first positioning and the second positioning is greater than a distance threshold;

[0012] If the distance between the first positioning and the second positioning is greater than a distance threshold, determining a final positioning of the person in the target area by combining the first positioning weight, the second positioning weight, the first positioning, the second positioning and the identity information;

[0013] If the distance between the first positioning and the second positioning is less than a distance threshold, the final positioning of the person in the target area is determined by combining the first positioning and the identity information.

[0014] Furthermore, the method for multimodal indoor positioning of personnel in the clean room also includes:

[0015] If the distance between the person in the target area and the image acquisition device is greater than a first threshold, and the second frequency is greater than the first frequency, the final location of the person in the target area is determined by combining the third positioning weight, the fourth positioning weight, the first positioning, the second positioning and the identity information.

[0016] Further, the image acquisition device acquires image information in the target area at a first frequency to obtain a first location of a person in the target area in a first time period, specifically:

[0017] The image acquisition device acquires image information in the target area at a first frequency, and obtains a target detection frame according to the image information;

[0018] The midpoint of the bottom edge of the target detection frame is used as the first marking point, and the midpoint of the line connecting the key points of the ankles of the person in the target detection frame is used as the second marking point;

[0019] The abscissa of the first marking point and the ordinate of the second marking point are integrated to obtain a first location of the person in the target area within a first time period.

[0020] Furthermore, the method for multimodal indoor positioning of personnel in the clean room also includes:

[0021] Combining the fused first positioning with internal parameters of the image acquisition device to obtain a first corrected positioning;

[0022] After the first corrected positioning is subjected to denormalization processing, the corrected first positioning is obtained in combination with external parameters of the image acquisition device.

[0023] Furthermore, the personnel in the clean room also wear a marking image;

[0024] The image acquisition device acquires image information in the target area at a first frequency to obtain a first location and a first person label of a person in the target area who is wearing a marking pattern in a first time period;

[0025] According to the first personnel tag, a first UWB tag corresponding to the first personnel tag is determined, the UWB base station collects a second signal of the first UWB tag at a second frequency, and obtains a second location of the person corresponding to the first personnel tag according to the second signal;

[0026] The final location of the person in the target area is determined by combining the first location, the second location and the first person tag.

[0027] In a second aspect, the present invention further provides a method for displaying multimodal indoor positioning of personnel in a clean room, comprising:

[0028] Obtaining a positioning display interface of the clean room, wherein the positioning display interface includes an activity area for personnel in the clean room and an electronic fence area for multiple experimental equipment;

[0029] Obtaining the final location of the personnel according to the multi-modal indoor positioning method for personnel in the clean room described in the first aspect;

[0030] If the final location is within the electronic fence area, adjusting the horizontal coordinate or vertical coordinate of the final location to the same horizontal coordinate or vertical coordinate as the nearest edge of the electronic fence area to obtain an adjusted final location;

[0031] In combination with the adjusted final location and the position information of the electronic fence area, the adjusted final location of the personnel in the clean room is displayed on the location display interface.

[0032] Furthermore, the electronic fence area of ​​the experimental device is specifically an electronic fence area with an edge of the experimental device as an edge, or an area with a distance from a center point of the experimental device less than a first distance as the electronic fence area;

[0033] The coordinate information of the electronic fence area includes coordinate information of each edge of the electronic fence area.

[0034] Furthermore, the final location set of personnel in multiple consecutive time periods within a preset time is obtained;

[0035] The personnel activity paths are obtained by connecting the personnel final location sets in chronological order.

[0036] In a third aspect, the present invention further provides a device for multimodal indoor positioning of personnel in a clean room, characterized in that a plurality of UWB base stations and a plurality of image acquisition devices are arranged in the clean room, the image acquisition device is arranged on the top of the clean room, the personnel in the clean room wear UWB tags, the transmission power of the UWB base station and the UWB tag is lower than the first transmission power, and the device comprises:

[0037] An image positioning module, used for the image acquisition device to acquire image information in the target area at a first frequency to obtain a first position of a person in the target area within a first time period;

[0038] A UWB positioning module, which is used for the UWB base station to collect the first signal of the UWB tag in the target area at the second frequency, and obtain the second positioning and identity information of the person in the target area within the first time period according to the first signal;

[0039] The positioning fusion module is used to determine the final positioning of the person in the target area by combining the first positioning, the second positioning and the identity information.

[0040] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the methods for multimodal indoor positioning of clean room personnel in the first aspect.

[0041] In a fifth aspect, the present invention further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it executes any one of the methods for multimodal indoor positioning of personnel in a clean room in the first aspect.

[0042] The beneficial effects of adopting the above technical solution are: the position of the visual positioning system is calibrated. Through the precise positioning of the UWB signal, the position and posture of the camera can be calibrated to improve the accuracy of the visual positioning system. The UWB positioning and camera visual information are integrated, and the UWB positioning information is used to assist the camera visual information, which can improve the positioning accuracy and robustness of the tracking system for the target. When the target crosses the field of view of different cameras, the UWB positioning information can be used to seamlessly switch and connect the target position. Carrying a UWB tag device on the target to be tracked, the target's position information can be obtained in real time using UWB technology. Even if the target is in the blind spot of the camera, or multiple people overlap and block the camera area, the approximate position information of the target can be obtained through UWB technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0044] Figure 1 This is a schematic diagram of a method for multi-modal indoor positioning of personnel in a clean room in one embodiment of the present application;

[0045] Figure 2 This is a schematic diagram of personnel location recognition by a clean room image acquisition device in one embodiment of the present application;

[0046] Figure 3 This is a schematic diagram of an internal parameter calibration board of a clean room image acquisition device in one embodiment of the present application;

[0047] Figure 4 This is a schematic diagram of various forms of internal parameter calibration plates of a clean room image acquisition device in one embodiment of the present application;

[0048] Figure 5 This is a schematic diagram of a clean room image acquisition device in one embodiment of the present application performing data acquisition in combination with external parameters;

[0049] Figure 6 This is a schematic diagram of a multi-modal indoor positioning display of a clean room personnel in one embodiment of the present application;

[0050] Figure 7 It is a schematic diagram of a device for multi-modal indoor positioning of personnel in a clean room in one embodiment of the present application. DETAILED DESCRIPTION

[0051] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In order to explain the present invention in more detail, the method and device for multimodal indoor positioning of clean room personnel provided by the present invention are specifically described below in conjunction with the drawings.

[0052] Unless otherwise defined, the technical terms or scientific terms used in the disclosure of this application should be understood by people with ordinary skills in the field to which the present invention belongs. The "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, similar words such as "one", "one" or "the" do not indicate a quantitative limit, but indicate that there is at least one. Similar words such as "include" or "include" mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Similar words such as "connect" or "connected" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship can also change accordingly.

[0053] When personnel enter a semiconductor clean room, they usually need to wear masks, glasses, sanitary caps and outer robes, etc. Most of the facial features or other characteristic information with obvious identifiable identity of the personnel are blocked, resulting in video personnel positioning only being able to identify the activities of people in the clean room and the specific location of the people. However, due to the inability to obtain facial feature information or other characteristic information with obvious differences in the personnel, the video personnel positioning technology in the clean room cannot determine the specific identity information of the personnel. In the absence of specific personnel information, continuous tracking of personnel across cameras over a large range cannot be achieved.

[0054] At the same time, since there are many large devices in the semiconductor clean room, most of the large devices are equipped with metal casings, and the floors, ceilings, walls, etc. of the semiconductor clean room are also mostly made of metal. Metal products will cause serious absorption, reflection, refraction and scattering of wireless signals, which will eventually lead to multipath effects, seriously reducing the accuracy and reliability of wireless positioning technology.

[0055] In this regard, the present invention provides a method for multi-modal indoor positioning of personnel in a clean room. By combining video personnel positioning technology and ultra-wideband positioning technology, personnel in a clean room with a large amount of metal interference can be accurately and efficiently positioned. This method is applied to a terminal device as an example for illustration. Figure 1 A schematic diagram showing a method for multimodal indoor localization of clean room personnel.

[0056] The present application embodiment provides an application scenario of a method for indoor positioning of multimodal personnel in a clean room, and the application scenario includes a terminal device provided in the embodiment, and the terminal device includes but is not limited to a smart phone and a computer device, wherein the computer device can be at least one of a desktop computer, a portable computer, a laptop computer, a mainframe computer, a tablet computer, etc. The user operates the terminal device to obtain accurate positioning of the personnel in the clean room. For the specific process, please refer to the embodiment of the method for indoor positioning of multimodal personnel in a clean room.

[0057] In this embodiment, the method for multi-modal indoor positioning of personnel in a clean room is applied to a clean room positioning system. The clean room positioning system is arranged inside the clean room and is equipped with multiple ultra-wideband base stations, multiple image acquisition devices and a host computer.

[0058] Among them, ultra-wideband positioning technology (UWB) is a technology that uses ultra-wideband wireless signals to achieve positioning. The communication frequency of ultra-wideband wireless signals is usually greater than 500MHz, and has the advantages of strong signal propagation capability and penetration capability and a large positioning range. In this embodiment, the location and number of specific ultra-wideband base stations can be set according to the specific layout of the clean room and the distribution of metals to ensure that the ultra-wideband base stations can evenly cover the entire laboratory area and reduce signal blind spots; further, since ultra-wideband signals are easily interfered by metal, resulting in positioning deviations, in order to ensure the communication quality between the ultra-wideband base station in the clean room and the ultra-wideband tag worn by the personnel, in this embodiment, the transmission power of the ultra-wideband base station and the ultra-wideband tag worn by the personnel is reduced to less than the first transmission power, thereby reducing the influence of metal reflection and scattering on the ultra-wideband signal during transmission. At this time, the ultra-wideband signal communication can achieve a balanced state with optimal signal strength and communication quality and minimal metal interference in the clean room environment. In addition, in order to reduce direct contact between the ultra-wideband base station and metal objects in the clean room, the ultra-wideband base station is installed in the clean room using a bracket to reduce the interference of metal on the ultra-wideband signal, and the transmission path of the ultra-wideband base station signal can be optimized by adjusting the height and angle of the bracket.

[0059] For the image acquisition device in the clean room, the present embodiment can set the position, angle, height and number of the image acquisition device according to the specific image acquisition requirements, so as to reduce the blind spot of the field of vision in the clean room. In the present embodiment, the image acquisition device can use an external camera or a monitoring camera in the clean room to capture images of personnel, and the above camera is set at the ceiling position of the clean room.

[0060] The above-mentioned image acquisition device and ultra-wideband base station are both connected to the host computer for communication, and the image information of the image acquisition device and the ultra-wideband signal received by the ultra-wideband base station are transmitted to the host computer for processing to obtain accurate positioning of the personnel in the clean room.

[0061] Corresponding to the ultra-wideband base station in the above-mentioned clean room positioning system, personnel entering the clean room wear an ultra-wideband tag. The ultra-wideband tag can be set to a personnel card or bracelet mode. Specific personnel identity information, such as the personnel ID and name of the person entering the clean room, can be identified based on the ultra-wideband tag worn by the person.

[0062] Based on the above-mentioned clean room positioning system, the multi-modal indoor positioning method for clean room personnel in this embodiment includes:

[0063] Step S101: an image acquisition device acquires image information in a target area at a first frequency to obtain a first position of a person in the target area within a first time period.

[0064] Specifically, the first location of the person in step S101 is the location coordinates of the person, which can be recorded as In order to obtain the positioning coordinates of the personnel closer to the actual position of the personnel, this embodiment adopts two key point detection methods to obtain the positioning of the personnel, including the following steps:

[0065] Step S201: an image acquisition device acquires image information in a target area at a first frequency, and obtains a target detection frame according to the image information.

[0066] The image acquisition device can obtain image information in the target area, and the image information includes image information of the person to be located. At this time, by detecting the target of the image information, a target detection frame including the person to be located is obtained. The target detection frame can be used to describe the position information of the person, and is usually represented by a rectangle. In this embodiment, a variety of target detection methods in existing means can be used to obtain the target detection frame in the image information, and no further limitation is made to this.

[0067] Step S202: The midpoint of the bottom edge of the target detection frame is used as the first marking point, and the midpoint of the line connecting the key points of the ankles of the person in the target detection frame is used as the second marking point.

[0068] Among them, personnel positioning can usually be done by marking two key points, as shown in the following figure: Figure 2 As shown in the figure, one method uses the midpoint of the bottom edge of the target detection box (i.e., the Figure 2 The red dot in the middle is used as the personnel positioning. Another method is to use the midpoint of the line connecting the two ankle key points (i.e. the Figure 2 The blue dot in the middle is used as the personnel location. However, most cameras in clean rooms are set on the ceiling of the clean room, and the personnel in the clean room are all shot from a bird's-eye view. Therefore, the personnel location obtained by the above two key point marking methods is easy to deviate from the actual position of the personnel.

[0069] Step S203: fusing the horizontal coordinate of the first marking point and the vertical coordinate of the second marking point to obtain a first location of the person in the target area within a first time period.

[0070] In order to obtain personnel positioning information that is closer to the actual position, this embodiment converts the first marking point obtained by the above two methods into and the second marker To blend, take the horizontal coordinate of the first marked point As the abscissa after fusion, take the ordinate of the second marker point As the fused vertical coordinate, we get the first fused positioning. .

[0071] Furthermore, the first position after fusion mentioned above is only the pixel coordinates of the camera image. However, the pixel coordinates need to be projected onto the building plane of the clean room to obtain accurate physical coordinates in order to obtain the first position of the person captured by the camera. Considering the heterogeneity of data collected by different cameras, the videos captured by different cameras may have different resolutions, viewing angles, lighting conditions, etc., or the camera may be distorted. Therefore, it is necessary to combine the internal and external parameters of the camera to fuzzily calculate the first position after fusion. Make corrections, specifically:

[0072] Step S301, combining the fused first positioning and the internal parameters of the camera to obtain a first corrected positioning. The specific expression of this step can be expressed as: , For the first positioning after fusion, is the internal parameter of the camera, is the first calibration parameter.

[0073] Step S302, after performing denormalization processing on the first correction position, the first position of the person on the clean room plane is obtained by combining the external parameters of the camera. The specific expression of this step is: ,in The first correction position for the denormalization process is , and the expression between the first correction parameter is , is the first external parameter of the camera, is the second external parameter of the camera, is the first positioning on the clean room plane. In this embodiment, the first positioning of the personnel on the clean room plane is recorded as .

[0074] Step S102: The UWB base station collects a first signal of a UWB tag in a target area at a second frequency, and obtains second positioning and identity information of a person in the target area within a first time period according to the first signal.

[0075] Specifically, the host computer records the location information of the UWB base station on the clean room plane, establishes a communication connection by scanning the UWB tags in the target area through the UWB base station, and sends a signal to the UWB tag. The second location of the person wearing the UWB tag is calculated according to the first signal returned by the UWB tag, which is recorded as Among them, the UWB tag worn by the personnel contains the personnel's specific identity information, including but not limited to the personnel ID and name of the personnel entering the clean room. The identity information carried in the UWB tag can provide identification information for overlapping image acquisition or cross-camera tracking in the clean room to locate the personnel.

[0076] Step S103, determining the final location of the person in the target area by combining the first location, the second location and the identity information.

[0077] The final location of the personnel is determined by fusing the location information obtained by the above multi-modal means. The same time period is used in the above first location and second location determination process to ensure that when the acquisition frequencies of the image acquisition device and the UWB base station are different (i.e., the first frequency is not equal to the second frequency), the location points closest in time can be fused. At this time, the final location obtained is closest to the actual location of the personnel. In this regard, step S103 includes the following specific steps:

[0078] Step S401: if the distance between the person in the target area and the image acquisition device is less than a first threshold, determine whether the first positioning With the second positioning Whether the distance between them is greater than the distance threshold.

[0079] Specifically, a precise acquisition range can be defined according to specific performance parameters of the image acquisition device. The precise acquisition range is a range defined with the plane coordinates of the image acquisition device as the center and a first threshold as the radius. When a person enters the precise acquisition range, the distance between the person and the image acquisition device is less than the first threshold. At this time, the first positioning obtained by the image acquisition device has a higher confidence level.

[0080] Step S402: If the distance between the first positioning and the second positioning is greater than a distance threshold, the final positioning of the person in the target area is determined by combining the first positioning weight, the second positioning weight, the first positioning, the second positioning and the identity information.

[0081] Specifically, when both image acquisition and UWB positioning technologies have high confidence, if there is a large difference between the positioning results of the two, the final positioning of the person obtained by fusing the two technologies has a higher reference value, specifically: , ,in is the horizontal coordinate of the final positioning, is the ordinate of the final positioning, It is the ID of the clean room personnel. Indicates daily time period, is the first positioning weight, is the second positioning weight, and the final positioning of the personnel can be recorded as .

[0082] Step S403: If the distance between the first positioning and the second positioning is less than a distance threshold, the final positioning of the person in the target area is determined by combining the first positioning and the identity information.

[0083] Specifically, when a person is in the area of ​​accurate image acquisition, if the positioning results of image acquisition and UWB positioning technology are not much different, the positioning obtained by one of the technologies can be directly used as the final positioning. In this embodiment, considering that UWB positioning technology is still slightly affected by the metal environment, it is preferred to use the positioning result obtained by image acquisition. , , the final position of the personnel is recorded as .

[0084] Step S404: If the distance between the person in the target area and the image acquisition device is greater than the first threshold, and the second frequency is greater than the first frequency, the final positioning of the person in the target area is determined by combining the third positioning weight, the fourth positioning weight, the first positioning, the second positioning and the identity information.

[0085] Specifically, when the person is not within the precise image acquisition range, the difference between the image acquisition frequency and the UWB acquisition frequency will affect the accuracy, and it is necessary to evaluate the first positioning of image acquisition and the second positioning of UWB acquisition in combination with the specific location information; at this time, based on the limitations of computing resources and algorithm performance, as long as the UWB acquisition frequency is greater than the image acquisition frequency, it will not affect the accuracy, so that the accuracy of the positioning fusion of the two technologies can be guaranteed. At this time, the expression is: , , is the horizontal coordinate of the final positioning, is the ordinate of the final positioning, It is the ID of the clean room personnel. Indicates daily time period, is the third positioning weight, is the fourth positioning weight, and the final positioning of the personnel can be recorded as .

[0086] Based on the above embodiment, considering that there are multiple personnel in the target area, in order to more quickly identify the personnel and align and merge the two technical positioning, the personnel in the clean room also wear a marking pattern. The multimodal positioning method of the clean room personnel of the present application also includes:

[0087] Step S501: an image acquisition device acquires image information in a target area at a first frequency to obtain a first location and a first person label of a person in the target area who wears a marking pattern in a first time period.

[0088] Specifically, a marking pattern is set on the top or shoulder of the sanitary cap of the personnel in the clean room, and the pattern can be a personnel ID pattern or other simple graphics with identification for distinguishing personnel; the marking pattern corresponds to the personnel ID one by one, for example, the rectangular pattern corresponds to personnel A, the circular pattern corresponds to personnel B, the star pattern corresponds to personnel C, etc. The first positioning obtained according to the above step S501 can be recorded as ,in This is the clean room personnel ID.

[0089] Step S502: According to the first personnel tag, a first UWB tag corresponding to the first personnel tag is determined, the UWB base station collects a second signal of the first UWB tag at a second frequency, and obtains a second location of the person corresponding to the first personnel tag according to the second signal.

[0090] Specifically, since the UWB tag worn by a person also corresponds to the person ID, the first positioning acquired by the image and the second positioning acquired by the UWB technology can be connected through the same person ID. When there are multiple people walking in the clean room, the positioning of the people can be quickly achieved. Among them, the second positioning of the person can be recorded as ,in This is the clean room personnel ID.

[0091] Step S503: Determine the final location of the person in the target area by combining the first location, the second location and the first person tag.

[0092] The specific process of fusing the positioning results of the image acquisition technology and the UWB positioning technology in step S503 can be found in the description of the above step S103. Since the fusing process of the two is the same, it will not be described here.

[0093] For the internal and external parameters of the image acquisition device considered in the above embodiments, Figure 3 , Attachment Figure 4 and attached Figure 5 As shown, the specific determination process is as follows:

[0094] The internal parameter determination process of the image acquisition device used in this embodiment is as follows:

[0095] As attached Figure 3 As shown, a person holds the calibration plate and finds a suitable distance so that the calibration plate occupies about 1 / 6 of the camera screen;

[0096] A person holds the calibration plate and moves it so that it moves in an S shape in the camera image, and gradually covers the entire camera image during the movement.

[0097] As attached Figure 4 As shown, the calibration board has 5 postures in the camera screen, and operation b is repeated in each state;

[0098] Each camera completes the above three processes and records the screen.

[0099] About 50 calibration images are extracted from the screen recording video of each camera (about 10 images for each calibration plate posture), and the internal parameters K of the camera are obtained using the calibration tool (opencv / matlab).

[0100] The external parameter determination process of the image acquisition device used in this embodiment is as follows:

[0101] Confirm the global coordinate origin and measure the coordinates of the local landmark points relative to the global origin;

[0102] As attached Figure 5 As shown in the figure, four marking points are obtained by measuring near the local landmark point, and corresponding marks are made with black and white grid paper. In order to facilitate subsequent processing, the four marking points should be as square or rectangular as possible if conditions permit;

[0103] Based on the physical coordinates and pixel coordinates of the four points, the calibration tool (opencv / matlab) is used to complete the calibration of the first external parameter R and the second external parameter t of the camera.

[0104] Based on the above embodiments of personnel positioning, combined with the plane interface of the clean room, the personnel positioning in the clean room is displayed. Figure 6 As shown, this embodiment also discloses a method for multi-modal indoor positioning display of clean room personnel, comprising the following steps:

[0105] Step S601, obtaining a positioning display interface of the clean room, wherein the positioning display interface includes an activity area for personnel in the clean room and an electronic fence area for multiple experimental equipment.

[0106] Among them, the electronic fence area of ​​the experimental equipment is specifically an electronic fence area with the edge of the experimental equipment as the edge, or an area with a distance from the center point of the experimental equipment less than the first distance as the electronic fence area; the coordinate information of the electronic fence area includes the coordinate information of each edge of the electronic fence area, and the electronic fence area is mostly displayed in a rectangle, and only the coordinates of the upper left corner and the lower right corner of the electronic fence can be recorded to display the specific positioning of the electronic fence, which can be recorded in this embodiment as and .

[0107] It should be noted that since the experimental equipment in the clean room is completely closed and personnel cannot enter, the electronic fence area of ​​the experimental equipment in the positioning display interface is equivalent to the prohibited area for personnel. In contrast to the electronic fence area of ​​the experimental equipment, the personnel activity area refers to the area in the clean room that personnel can enter, and is also the area where personnel positioning is displayed in the positioning display interface.

[0108] Step S602: Obtain the final location of the person .

[0109] Step S603: If the final location is within the electronic fence area, the horizontal coordinate or vertical coordinate of the final location is adjusted to be the same as the horizontal coordinate or vertical coordinate of the nearest edge of the electronic fence area to obtain an adjusted final location.

[0110] Specifically, if the final positioning Being inside the electronic fence means that the final positioning may be affected by the metal environment and may deviate. In this case, the final positioning needs to be adjusted based on the coordinates of the edge of the electronic fence area, for example:

[0111] (1) When and When the final position is 0, it is necessary to further determine the distance between the final position and the edge of the electronic fence area, so as to adjust both the horizontal and vertical coordinates of the final position. The specific adjustment rules are as follows:

[0112] (a) If ,but ;

[0113] (b) If ,but ;

[0114] (c) If ,but ;

[0115] (d) If ,but .

[0116] (2) When , or When the final positioning is performed, it is necessary to further determine the distance between the final positioning and the edge of the electronic fence area in order to adjust the horizontal coordinate of the final positioning. The specific adjustment rules are as follows:

[0117] (a) If ,but ;

[0118] (b) If ,but .

[0119] (3) When , or When the distance between the final position and the edge of the electronic fence area needs to be further determined, the vertical coordinate of the final position can be adjusted. The specific adjustment rules are as follows:

[0120] (a) If ,but ;

[0121] (b) If ,but .

[0122] Step S604: combining the adjusted final location and the location information of the electronic fence area, the adjusted final location of the personnel in the clean room is displayed on the location display interface.

[0123] In addition, in the actual use of the above-mentioned personnel positioning technology, the inventors found through comparison of a large amount of data that there are many narrow passages in the clean room. Due to the accuracy of the image acquisition device and UWB positioning technology, personnel positioning is prone to deviation. The deviation is generally controlled within 2-3 meters. However, the width of the narrow passage in the clean room is mostly about 2 meters. When deviation occurs, it is easy for personnel to be located in the electronic fence area of ​​the experimental equipment or in the area outside the clean room. At this time, the final positioning of the personnel also needs to be adjusted, specifically:

[0124] When the personnel activity area in the clean room is a narrow passage, if the abscissa of the final positioning exceeds the rated range of the abscissa of the activity area, the abscissa of the midpoint of the line connecting the two edges of the narrow passage is used as the abscissa of the final positioning.

[0125] According to the plane display interface of the clean room, the coordinate information of the narrow channel area in the clean room is expressed as ; If the horizontal coordinate of the final positioning exceeds the narrow channel area, that is or When the horizontal coordinate of the final positioning is adjusted, the horizontal coordinate of the midpoint of the line perpendicular to the two edges of the narrow road is taken as the adjusted horizontal coordinate of the final positioning, that is, .

[0126] Similarly, if the two edges of the narrow lane define the narrow lane area through the longitudinal coordinate, it is necessary to compare the longitudinal coordinate of the final positioning with the rated range of the longitudinal coordinate of the narrow lane, and adjust the longitudinal coordinate of the final positioning.

[0127] Furthermore, since most of the experimental equipment in the clean room is large and high, it is easy to have blind spots in image acquisition. When a person enters the blind spot (equivalent to the image acquisition device being unable to obtain image information), the second positioning obtained by UWB technology is used as the final positioning of the person.

[0128] Based on this, the embodiment of the present application also proposes a method for determining the activity path of personnel in the clean room, specifically:

[0129] Step S701, obtaining a set of personnel final locations for a plurality of consecutive time periods within a preset time;

[0130] Step S702: Connect the personnel's final location sets in chronological order to obtain the personnel's activity path.

[0131] Among them, the preset time can be set to one day, 12 hours, 9 hours or 6 hours according to viewing needs; the preset time includes multiple consecutive time periods, and the interval of each time period can be determined according to the path accuracy, for example, set to 5 minutes, 3 minutes or 1 minute.

[0132] Furthermore, in order to distinguish the paths of different personnel in the clean room, different colors can be used to display the personnel's activity paths according to the personnel ID, so that the monitoring personnel can quickly determine the specific activity paths of the personnel from the monitoring interface.

[0133] The above-mentioned multi-modal positioning method for clean room personnel has the following advantages: 1. Calibrate the position of the visual positioning system. Through the precise positioning of the UWB signal, the position and posture of the camera can be calibrated to improve the accuracy of the visual positioning system. 2. Fusion of UWB positioning and camera visual information, using UWB positioning information to assist camera visual information, can improve the positioning accuracy and robustness of the tracking system for the target. When the target crosses the field of view of different cameras, the UWB positioning information can be used to seamlessly switch and connect the target position. 3. Carry a UWB tag device on the target to be tracked, and use UWB technology to obtain the target's location information in real time. Even if the target is in the blind spot of the camera, or multiple people overlap and block the camera area, the approximate location information of the target can be obtained through UWB technology.

[0134] It should be understood that although the Figure 1The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Figure 1 At least part of the steps may include multiple sub-steps or sub-stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0135] The embodiments disclosed in the present invention describe in detail the method for multi-modal indoor positioning of personnel in a clean room. The method disclosed in the present invention can be implemented using various devices. Therefore, the present invention also discloses a device for multi-modal indoor positioning of personnel in a clean room. Figure 7 , specific embodiments are given below to explain in detail.

[0136] An image positioning module 801 is used for the image acquisition device to acquire image information in the target area at a first frequency to obtain a first position of a person in the target area within a first time period;

[0137] The UWB positioning module 802 is used for the UWB base station to collect the first signal of the UWB tag in the target area at the second frequency, and obtain the second positioning and identity information of the person in the target area in the first time period according to the first signal;

[0138] The positioning fusion module 803 is used to determine the final positioning of the person in the target area by combining the first positioning, the second positioning and the identity information.

[0139] For the device of multimodal indoor positioning of personnel in the clean room, all of them can be referred to the above definition of the method, which will not be repeated here. Each module in the above device can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor of the terminal device in the form of hardware, or stored in the memory of the terminal device in the form of software, so that the processor calls and executes the operations corresponding to the above modules.

[0140] In one embodiment, the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method for multi-modal indoor positioning of personnel in a clean room are implemented.

[0141] The computer-readable storage medium may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM (erasable programmable read-only memory), a hard disk or a ROM. Optionally, the computer-readable storage medium includes a non-transitory computer-readable storage medium. The computer-readable storage medium has a storage space for program codes for executing any method step of the above method. These program codes may be read from or written into one or more computer program products, and the program codes may be compressed in an appropriate form.

[0142] In one embodiment, the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the above-mentioned method for multi-modal indoor positioning of personnel in a clean room when executing the computer program.

[0143] The computer device includes a memory, a processor, and one or more computer programs, wherein the one or more computer programs can be stored in the memory and configured to be executed by one or more processors, and the one or more application programs are configured to execute the above-mentioned method for multimodal indoor positioning of clean room personnel.

[0144] The processor may include one or more processing cores. The processor uses various interfaces and lines to connect the various parts of the entire computer device, and executes various functions of the computer device and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Optionally, the processor can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor can integrate one or more combinations of a central processing unit (Central Processing Unit, CPU), a reporting verifier (Graphics Processing Unit, GPU) for buried data, and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor, but may be implemented separately through a communication chip.

[0145] The memory may include a random access memory (RAM) or a read-only memory (ROM). The memory may be used to store instructions, programs, codes, code sets or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc. The data storage area may also store data created by the terminal device during use, etc.

[0146] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for multi-modal indoor positioning of personnel in a clean room, characterized in that: The clean room is provided with a plurality of UWB base stations and a plurality of image acquisition devices, the image acquisition devices are provided on the top of the clean room, personnel in the clean room wear UWB tags, the transmission power of the UWB base stations and the UWB tags is lower than the first transmission power, and the method comprises: The image acquisition device acquires image information in the target area at a first frequency to obtain a first location of a person in the target area within a first time period; The UWB base station collects a first signal of a UWB tag in a target area at a second frequency, and obtains second positioning and identity information of a person in the target area in a first time period according to the first signal; Determine the final location of the person in the target area by combining the first location, the second location and the identity information; The determining of the final location of the person in the target area by combining the first location, the second location and the identity information is specifically: If the distance between the person in the target area and the image acquisition device is less than a first threshold, determining whether the distance between the first positioning and the second positioning is greater than a distance threshold; If the distance between the first positioning and the second positioning is greater than a distance threshold, determining a final positioning of the person in the target area by combining the first positioning weight, the second positioning weight, the first positioning, the second positioning and the identity information; If the distance between the first location and the second location is less than a distance threshold, determine the final location of the person in the target area by combining the first location and the identity information; The image acquisition device acquires image information in the target area at a first frequency to obtain a first location of a person in the target area within a first time period, specifically: The image acquisition device acquires image information in the target area at a first frequency, and obtains a target detection frame according to the image information; The midpoint of the bottom edge of the target detection frame is used as the first marking point, and the midpoint of the line connecting the key points of the ankles of the person in the target detection frame is used as the second marking point; The abscissa of the first marking point and the ordinate of the second marking point are integrated to obtain a first location of the person in the target area within a first time period.

2. The method for multi-modal indoor positioning of personnel in a clean room as claimed in claim 1, characterized in that: Also includes: If the distance between the person in the target area and the image acquisition device is greater than a first threshold, and the second frequency is greater than the first frequency, the final location of the person in the target area is determined by combining the third positioning weight, the fourth positioning weight, the first positioning, the second positioning and the identity information.

3. The method for multi-modal indoor positioning of clean room personnel as claimed in claim 1, characterized in that: Also includes: Combining the fused first positioning with internal parameters of the image acquisition device to obtain a first corrected positioning; After the first corrected positioning is subjected to denormalization processing, the corrected first positioning is obtained in combination with external parameters of the image acquisition device.

4. The method for multi-modal indoor positioning of clean room personnel as claimed in claim 1, characterized in that: Personnel in the clean room also wear marking images; The image acquisition device acquires image information in the target area at a first frequency to obtain a first location and a first person label of a person in the target area who is wearing a marking pattern in a first time period; According to the first personnel tag, a first UWB tag corresponding to the first personnel tag is determined, the UWB base station collects a second signal of the first UWB tag at a second frequency, and obtains a second location of the person corresponding to the first personnel tag according to the second signal; The final location of the person in the target area is determined by combining the first location, the second location and the first person tag.

5. A method for displaying multi-modal indoor positioning of personnel in a clean room, characterized in that: include: Obtaining a positioning display interface of the clean room, wherein the positioning display interface includes an activity area for personnel in the clean room and an electronic fence area for multiple experimental equipment; Obtaining the final positioning of the personnel according to the method for multimodal indoor positioning of personnel in a clean room as described in any one of claims 1 to 4; If the final location is within the electronic fence area, adjusting the horizontal coordinate or vertical coordinate of the final location to the same horizontal coordinate or vertical coordinate as the nearest edge of the electronic fence area to obtain an adjusted final location; In combination with the adjusted final location and the position information of the electronic fence area, the adjusted final location of the personnel in the clean room is displayed on the location display interface.

6. The method for displaying the multi-modal indoor positioning of personnel in a clean room according to claim 5, characterized in that: The electronic fence area of ​​the experimental device is specifically an electronic fence area with an edge of the experimental device as an edge, or an area with a distance from the center point of the experimental device less than a first distance as the electronic fence area; The coordinate information of the electronic fence area includes coordinate information of each edge of the electronic fence area.

7. The method for displaying the multi-modal indoor positioning of personnel in a clean room according to claim 5, characterized in that: Also includes: When the personnel activity area in the clean room is a narrow passage, if the abscissa of the final positioning exceeds the rated range of the abscissa of the activity area, the abscissa of the midpoint of the line connecting the two edges of the narrow passage is used as the abscissa of the final positioning.

8. The method for displaying the multi-modal indoor positioning of personnel in a clean room according to claim 5, characterized in that: Also includes: Obtain the final location set of personnel for multiple consecutive time periods within a preset time; The personnel activity paths are obtained by connecting the personnel final location sets in chronological order.

9. A multi-modal indoor positioning device for personnel in a clean room, characterized in that: The clean room is provided with a plurality of UWB base stations and a plurality of image acquisition devices, the image acquisition devices are provided on the top of the clean room, personnel in the clean room wear UWB tags, the transmission power of the UWB base station and the UWB tag is lower than the first transmission power, and the device comprises: An image positioning module, used for the image acquisition device to acquire image information in the target area at a first frequency to obtain a first position of a person in the target area within a first time period; A UWB positioning module, which is used for the UWB base station to collect the first signal of the UWB tag in the target area at the second frequency, and obtain the second positioning and identity information of the person in the target area within the first time period according to the first signal; A positioning fusion module, used to determine the final position of the person in the target area by combining the first positioning, the second positioning and the identity information; The determining of the final location of the person in the target area by combining the first location, the second location and the identity information is specifically: If the distance between the person in the target area and the image acquisition device is less than a first threshold, determining whether the distance between the first positioning and the second positioning is greater than a distance threshold; If the distance between the first positioning and the second positioning is greater than a distance threshold, determining a final positioning of the person in the target area by combining the first positioning weight, the second positioning weight, the first positioning, the second positioning and the identity information; If the distance between the first location and the second location is less than a distance threshold, determine the final location of the person in the target area by combining the first location and the identity information; The image acquisition device acquires image information in the target area at a first frequency to obtain a first location of a person in the target area within a first time period, specifically: The image acquisition device acquires image information in the target area at a first frequency, and obtains a target detection frame according to the image information; The midpoint of the bottom edge of the target detection frame is used as the first marking point, and the midpoint of the line connecting the key points of the ankles of the person in the target detection frame is used as the second marking point; The abscissa of the first marking point and the ordinate of the second marking point are integrated to obtain a first location of the person in the target area within a first time period.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for multimodal indoor positioning of clean room personnel in any one of claims 1 to 8 are implemented.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, it performs the method for multimodal indoor positioning of clean room personnel according to any one of claims 1-8.

Images