Inspection method and inspection system

Abstract

The invention provides an inspection method and an inspection system. The inspection method comprises the following steps: acquiring relative position information among a plurality of image capturing devices; based on the target event and the relative position information, an inspection route meeting the target event is determined, and a plurality of image capturing devices included in the inspection route are set as a plurality of inspection devices; and controlling the real-time video signal of each inspection device to be displayed to the display device based on the inspection route.

Description

Technical Field

[0001] The present invention relates to a monitoring method and system, and particularly to an inspection method and inspection system based on multiple imaging devices. Background Art

[0002] In a traditional monitoring system using multiple cameras, the images captured by these cameras are displayed separately. Therefore, it can only view fixed angles and fixed positions, and cannot connect the images of multiple cameras. In addition, due to the large number of cameras, there are too many presented images, and it is difficult for users to find the target object among the numerous images. When switching the images of different cameras, due to the viewing angle difference between cameras, users cannot obtain an immersive experience.

[0003] In addition, for a monitoring system using 360-degree cameras, its disadvantage is that the degree of freedom is too high, and users need to manually turn to find the target object. Even when using multiple 360-degree cameras, only the image of one camera can be selected to view separately. When viewing a specific target, the traditional monitoring system cannot find the specific target in a short time. Summary of the Invention

[0004] The present invention provides an inspection method and inspection system, which can condense the viewing of real-time video signals that meet the target event.

[0005] The inspection method of the present invention is suitable for being executed by an electronic device. The inspection method includes: obtaining relative position information between multiple imaging devices; determining an inspection route based on the target event and the relative position information, where the inspection route meets the target event, and multiple imaging devices passing through in the inspection route are set as multiple inspection devices; and controlling the real-time video signals of each inspection device to be presented on a display device based on the inspection route.

[0006] According to an embodiment of the present invention, the above inspection method further includes: establishing the relative position information between the imaging devices, including: providing a floor plan corresponding to the space where the imaging devices are arranged; based on user operations, marking multiple planar positions corresponding to the actual positions of the imaging devices arranged in the space in the floor plan; and calculating the relative position information between the imaging devices based on the planar positions.

[0007] According to an embodiment of the present invention, the above inspection method further includes: establishing the relative position information between the imaging devices, including: respectively obtaining multiple images corresponding to the imaging devices from the imaging devices; and calculating the relative position information between the imaging devices by finding corresponding feature points in every two images.

[0008] According to an embodiment of the present invention, the above-mentioned target event includes an event for indicating that a specified object is photographed. The step of determining the inspection route includes: performing a target detection algorithm on the real-time video signal received by each imaging device by executing an artificial intelligence (AI) model to determine whether the imaging device photographs the specified object; and in response to the multiple target devices in the imaging device photographing the specified object, determining an inspection device based on the relative position information and the target device that photographs the specified object, wherein the number of inspection devices included in the inspection route is greater than or equal to the number of target devices.

[0009] According to an embodiment of the present invention, after determining whether the imaging device photographs the specified object, it further includes: in response to only the first imaging device in the imaging device photographing the specified object, determining an inspection route based on the relative position information and the first imaging device that photographs the specified object, wherein the inspection route at least includes the first imaging device and a second imaging device corresponding to a predetermined position.

[0010] According to an embodiment of the present invention, after determining whether the imaging device photographs the specified object, it further includes: in response to the specified object being a device, after detecting the presence of a device in the real-time video signal by executing a target detection algorithm through an artificial intelligence model, obtaining the real-time information of the device and recording the real-time information. The step of controlling the real-time video signal of each inspection device to be presented on the display device further includes: in response to the presence of a specified object in the real-time video signal, when the real-time video signal is presented on the display device, presenting the corresponding real-time information on the display device at the same time.

[0011] According to an embodiment of the present invention, after determining whether the imaging device photographs the specified object, it further includes: in response to the specified object being a human body, after detecting the presence of a human body in the real-time video signal by executing a target detection algorithm through an artificial intelligence model, determining whether the human body is in a dangerous state through the artificial intelligence model, and recording a warning message when it is determined that the human body is in a dangerous state. The step of controlling the real-time video signal of each inspection device to be presented on the display device further includes: in response to the presence of a specified object in the real-time video signal and the specified object having a warning message, when the real-time video signal is presented on the display device, presenting the warning message on the display device at the same time.

[0012] According to an embodiment of the present invention, after determining whether the imaging device has captured a specified object, the method further includes: in response to the specified object being a human body, after detecting that there is a human body in the real-time video signal by executing a target detection algorithm through an artificial intelligence model, generating a selection box for framing the human body through the artificial intelligence model. The step of controlling the real-time video signals of each inspection device to be presented on the display device further includes: in response to the presence of a specified object in the real-time video signal, when the real-time video signal is presented on the display device, presenting the selection box in the display device to frame the human body at the same time.

[0013] According to an embodiment of the present invention, the above step of controlling the real-time video signals of each inspection device to be presented on the display device based on the inspection route includes: switching the display screen of the display device from the real-time video signal of the first inspection device in the inspection device to the real-time video signal of the second inspection device that has captured the specified object in the inspection device, which includes: controlling the first inspection device to turn to the first direction towards the second inspection device for imaging, and controlling the second inspection device to turn to the first direction; presenting the real-time video signal of the first inspection device in the first direction on the display screen; performing a zoom operation on the real-time video signal of the first inspection device in the display screen; and after performing the zoom operation, controlling the second inspection device to turn from the first direction towards the second direction of the specified object for imaging, and synchronously switching the display screen to the real-time video signal of the second inspection device during the turning process of the second inspection device.

[0014] According to an embodiment of the present invention, the above target event includes an event for indicating the inspection of at least one working area. The step of determining the inspection route includes: selecting at least one target device corresponding to at least one working area in the imaging device; and determining the inspection device based on the relative position information and the at least one target device.

[0015] According to an embodiment of the present invention, the above inspection method further includes: determining the inspection devices included in the inspection route based on the target event and at least another target event; and determining the inspection order of the inspection devices with reference to the event order of the target event and at least another target event and based on the relative position information.

[0016] According to an embodiment of the present invention, the above step of determining the inspection route includes: determining the inspection order of the inspection devices based on the relative position information and the priority of the imaging device.

[0017] According to an embodiment of the present invention, in the process of sequentially displaying the video signals of each inspection device to a display device based on the inspection order, the following steps are further included: in response to detecting that a new event meets a target event, reselecting a plurality of imaging devices as a plurality of new inspection devices in the imaging device; based on the relative position information, taking the imaging device corresponding to the video signal currently displayed on the display device as the inspection starting point, and re-determining a new inspection route for the new inspection devices; and based on the new inspection route, controlling the real-time video signals of the new inspection devices to be presented to the display device.

[0018] According to an embodiment of the present invention, the above inspection method further includes: providing an inspection result interface to the display device, where the inspection result interface includes a video block, a floor plan block, an inspection screenshot block, and an information block. The video block is used to play the real-time video signal in real time. The floor plan block is used to display the floor plan corresponding to the space where the imaging device is located, and the floor plan includes a plurality of position information corresponding to the actual positions of the imaging devices set in the space in the floor plan and a trajectory based on the inspection order. The inspection screenshot block is used to display the screenshot corresponding to the target event. The information block is used to display the real-time information corresponding to the target event.

[0019] According to an embodiment of the present invention, in the process of controlling the real-time video signals of each inspection device to be presented to the display device based on the inspection route, in response to receiving a selection of a position in the real-time video signal presented on the display device, the real-time information corresponding to the specified object or work area included in the position is simultaneously presented on the display device.

[0020] The inspection system of the present invention includes: a plurality of imaging devices; a display device; and a processor coupled to the imaging devices and the display device. The processor is configured to: obtain the relative position information between the plurality of imaging devices; based on the target event and the relative position information, determine an inspection route, where the inspection route meets the target event, and the multiple imaging devices passed through in the inspection route are set as a plurality of inspection devices; and based on the inspection route, control the real-time video signals of the inspection devices to be presented to the display device.

[0021] Based on the above, the present invention provides an inspection method and an inspection system, which can select devices that meet the target event from a plurality of imaging devices, generate an inspection route based on this, and then display the content obtained by the imaging devices based on the inspection route. Accordingly, the viewing of the real-time video signals that meet the target event can be concentrated. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are included to provide a further understanding of the present invention, and are incorporated into and constitute a part of this specification. The drawings illustrate embodiments of the present invention and, together with the description, are used to explain the principles of the present invention.

[0023] Figure 1 It is a schematic diagram of an inspection system according to an embodiment of the present invention.

[0024] Figure 2 It is a schematic diagram of the architecture of an inspection system according to an embodiment of the present invention.

[0025] Figure 3 It is a flowchart of an inspection method according to an embodiment of the present invention.

[0026] Figure 4 It is a schematic diagram of a floor plan according to an embodiment of the present invention.

[0027] Figures 5A - 5C It is a schematic diagram of setting an inspection sequence according to an embodiment of the present invention.

[0028] Figure 6 It is a schematic diagram of setting an inspection route according to an embodiment of the present invention.

[0029] Figure 7 It is a schematic diagram of an inspection movement mode according to an embodiment of the present invention.

[0030] Figure 8 It is a schematic diagram of an inspection movement mode according to an embodiment of the present invention.

[0031] Figure 9 It is a schematic diagram of a display screen according to an embodiment of the present invention.

[0032] Figure 10 It is a schematic diagram of a display screen according to an embodiment of the present invention.

[0033] Figure 11 It is a schematic diagram of an inspection result interface according to an embodiment of the present invention.

[0034] Figure 12 It is a schematic diagram of a display screen according to an embodiment of the present invention.

[0035] Figure 13 It is a schematic diagram of a display screen according to an embodiment of the present invention.

[0036] Figure 14 It is a schematic diagram of an inspection route according to an embodiment of the present invention.

[0037] Figure 15 It is a schematic diagram of an inspection route according to an embodiment of the present invention.

[0038] Explanation of the reference numerals in the drawings

[0039] 100: Inspection system

[0040] 110: Processor

[0041] 120: Memory

[0042] 130: Display device

[0043] 140, 140-1 to 140-N, 4C1 to 4C5, 5C1 to 5C4, 6C1 to 6C5, 7C1 to 7C2, 8C1 to 8C3, 14C1 to 14C4, 15C1 to 15C2: Imaging device

[0044] 100A: Electronic device

[0045] 210: Streaming server

[0046] 220: Artificial intelligence model

[0047] 230: Receiving device

[0048] 241: Event server

[0049] 250: Patrol inspection module

[0050] 400: Floor plan

[0051] 71d, 72d, 81d, 82d, 83d, 14d1 to 14d6, 15d1 to 15d2: Direction

[0052] 900, 1000, 1200, 1300: Display screen

[0053] 910 to 930, 1210 to 1250: Text box

[0054] 1010 to 1040, 12F1 to 12F5, 12W1 to 12W2, 13W1 to 13W3: Selection box

[0055] 1100: Patrol inspection result interface

[0056] 1110: Video block

[0057] 1120: Floor plan block

[0058] 1130: Patrol inspection screenshot block

[0059] 1140: Information block

[0060] 1510, 1520: Equipment

[0061] T: Object

[0062] U, U1, U2, 14U1 to 14U4, 15U1 to 15U4: User

[0063] V2 to VN: Real-time video signal

[0064] S305 - S315: Steps of the inspection method Detailed implementation manners

[0065] Figure 1 is a schematic diagram of an inspection system according to an embodiment of the present invention. Please refer to Figure 1 , the inspection system 100 includes a processor 110, a memory 120, a display device 130, and N imaging devices 140-1 to 140-N (collectively referred to as the imaging devices 140), where N is an integer greater than or equal to 2. The processor 110 is coupled to the memory 120, the display device 130, and the imaging devices 140-1 to 140-N.

[0066] In this embodiment, the processor 110, the memory 120, and the display device 130 may be integrated in the same electronic device 100A. The electronic device 100A is, for example, a device with computing functions such as a smart phone, a tablet computer, a notebook computer, a personal computer, a vehicle navigation device, etc. The imaging devices 140-1 to 140-N are communicatively connected to the electronic device 100A through wired or wireless means, so that data can be transmitted between the imaging devices 140-1 to 140-N and the processor 110.

[0067] In another embodiment, it can also be set that the processor 110 and the memory 120 are integrated in the same electronic device with computing functions such as a smart phone, a tablet computer, a notebook computer, a personal computer, a vehicle navigation device, etc. The display device 130 and the imaging devices 140-1 to 140-N are communicatively connected to the electronic device through wired or wireless means.

[0068] The processor 110 is, for example, a Central Processing Unit (CPU), a Graphic Processing Unit (GPU), a Physics Processing Unit (PPU), a programmable microprocessor, an embedded control chip, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), or other similar devices.

[0069] The memory 120 is, for example, any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, hard disk, or other similar devices, or a combination of these devices. The memory 120 also includes one or more code segments, which, after being installed, are executed by the processor 110 to perform the subsequent inspection method.

[0070] The display device 130 is implemented, for example, using a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, a projection system, and the like.

[0071] The imaging devices 140-1 to 140-N are cameras, such as charge-coupled device (CCD) lenses, complementary metal oxide semiconductor transistors (CMOS) lenses, cameras, and the like. For example, the imaging devices 140-1 to 140-N are omnidirectional cameras. An omnidirectional camera (also known as a 360-degree camera) is a camera whose imaging perspective can cover the entire spherical surface or at least cover the annular field of view on the horizontal plane. Its types include full-sphere omnidirectional cameras and half-sphere omnidirectional cameras. In addition, the imaging devices 140-1 to 140-N can also be wide-angle cameras. In practical applications, multiple imaging devices 140-1 to 140-N are deployed in a space, and then a monitoring network is established based on the relationship between the imaging devices 140-1 to 140-N.

[0072] Figure 2 It is a schematic diagram of the architecture of an inspection system according to an embodiment of the present invention. Please refer to Figure 2 The inspection system 100 further includes a streaming server 210, an artificial intelligence (AI) model 220, a receiving device 230, an event server 240, and an inspection module 250. Here, the streaming server 210 is an independent server different from the electronic device 100A and is communicatively connected to the electronic device 100A via wired or wireless means. The streaming server 210 is used to store the real-time video signals of the imaging devices 140-1 to 140-N and transmit the real-time video signals to the inspection module 250.

[0073] The artificial intelligence model 220 is set in the memory 120 of the electronic device 100A as an application program composed of one or more code segments, and is executed through the processor 110 to execute an object detection algorithm on the real-time video signals received by each imaging device 140 through the artificial intelligence model 220, so as to determine whether the imaging device 140 captures a specified object. In addition, the artificial intelligence model 220 may also be set in another electronic device different from the electronic device 100A, and the other electronic device establishes a communication connection with the electronic device 100A through wired or wireless means.

[0074] The inspection module 250 is an application program stored in the memory 120 of the electronic device 100A and composed of one or more code segments, and is executed through the processor 110 to implement the inspection method described below.

[0075] The receiving device 230 is used to receive real-time information from the device and transmit the real-time information to the event server 240 for storage. The receiving device 230 may be a sensor or a programmable logic controller (PLC) provided in each device to monitor the operation status of the device in real time.

[0076] The event server 240 may be a database system provided in the electronic device 100A, which is used to store the real-time information transmitted by the receiving device 230 and store the recognition results of the artificial intelligence model 220. In addition, the event server 240 may also be an independent server different from the electronic device 100A, and communicates with the electronic device 100A through wired or wireless means. The event server 240 provides the recognition results of the artificial intelligence model 220 and / or the real-time information obtained by the receiving device 230 to the inspection module 250 according to the requirements of the inspection module 250.

[0077] The following describes each step of the inspection method in conjunction with the above inspection system 100. Figure 3 It is a flowchart of an inspection method according to an embodiment of the present invention. Please refer to Figures 1 - 3 In step S305, the processor 110 obtains the relative position information between a plurality of imaging devices 140-1 to 140-N.

[0078] In one embodiment, the processor 110 obtains a floor plan of the space where the imaging devices 140-1 to 140-N are disposed, and this floor plan includes a plurality of position information corresponding to the actual positions of the imaging devices 140-1 to 140-N in the space on the floor plan. Specifically, the processor 110 can display the floor plan corresponding to this space to the display device 130, and receive user operations through input devices such as a keyboard, a mouse, and a touchpad to mark the positions of the imaging devices 140-1 to 140-N on the floor plan. After that, the processor 110 obtains the relative position information between the imaging devices 140-1 to 140-N based on this position information.

[0079] For example, Figure 4 is a schematic diagram of a floor plan according to an embodiment of the present invention. In Figure 4 it, five imaging devices 4C1 to 4C5 are used for illustration. The floor plan 400 can be a simple floor plan or a space design diagram in formats such as DXF and DWG drawn by Computer Aided Design (CAD) software.

[0080] Please refer to Figure 4 , the user can manually set the planar positions of the imaging devices 4C1 to 4C5 in the space with respect to the floor plan 400 in the floor plan 400. For example, the imaging device 4C1 is set at the door, and the imaging device 4C2 is set at the corner of the entrance, etc. After determining the planar positions of the imaging devices 4C1 to 4C5 in the floor plan 400, the processor 110 can calculate the relative position information between the imaging devices 4C1 to 4C5 based on the planar positions of the imaging devices 4C1 to 4C5 in the floor plan 400. For example, which direction the imaging device 4C2 is located with respect to the imaging device 4C1.

[0081] In addition, the processor 110 can also automatically calculate the relative position information between the imaging devices 4C1 to 4C5 according to the images captured by the imaging devices 4C1 to 4C5 respectively. For example, the processor 110 obtains corresponding multiple images from the imaging devices 4C1 to 4C5 respectively (one imaging device captures one image), and calculates the relative position information between the imaging devices 4C1 to 4C5 by finding corresponding feature points in every two images. For example, assuming that the imaging ranges of the imaging device 4C1 and the imaging device 4C2 cover the same area, and the same feature values are found based on the two obtained images, the corresponding relationship between the two can be known. For example, which direction the imaging device 4C2 is located with respect to the imaging device 4C1.

[0082] The Scale-invariant feature transform (SIFT) method or the optical flow method can be used to find the feature points of the same target object in two images, and operations such as rotation, translation, zooming in, and zooming out are performed on the two images to match the feature points of the target object in the two images, so as to obtain the relative position information between the imaging device 4C1 and the imaging device 4C2. Perform perspective projection on each of the imaging devices 140 at 90 degrees in the front, back, left, and right directions, and then find the corresponding relationship between the images after these perspective projections. For example, the relative position information can be a homography transformation matrix between the two imaging devices. Through the homography transformation matrix, the included angle and the distance between the imaging device 4C1 and the imaging device 4C2 can be known, etc.

[0083] In addition, the above two methods can also be combined to obtain the relative position information. For example, after obtaining the relative position information using the planar graph, perform perspective projection using the corresponding angles of the method of marking the planar graph, and then make a comparison.

[0084] Next, in step S310, the processor 110 determines the inspection route based on the target event and the relative position information. Here, the determined inspection route will satisfy the target event, and multiple imaging devices passed by in the inspection route are set as multiple inspection devices. That is, the selected multiple inspection devices can satisfy the content of the target event. For example, the target event can be an event used to indicate that a specified object is photographed. For example, the specified object can be a human body, an animal, a plant, a household appliance, an electronic instrument, a device, a building material, etc. Or, the target event can also be an event used to indicate at least one working area (such as a test area, a production area, a packaging area). The processor 110 can further determine the inspection order of the inspection devices based on the relative position information and the priorities of the imaging devices 140-1 to 140-N. In other embodiments, the inspection order of the inspection devices can also be manually set by the user.

[0085] Taking the target event as an event used to indicate that a specified object is photographed, the processor 110 executes the artificial intelligence model 220 to perform a target detection algorithm on the real-time video signals received by each imaging device 140, and determines whether each imaging device 140 has photographed the specified object. In response to multiple target devices among the imaging devices 140-1 to 140-N photographing the specified object, multiple inspection devices are determined based on the relative position information and the target devices that have photographed the specified object. Here, the number of multiple inspection devices included in the inspection route is greater than or equal to the number of target devices.

[0086] In addition, in response to only one imaging device (the first imaging device) capturing a specified object, at least two inspection devices are determined based on the relative position information and the first imaging device that has captured the specified object. That is, the inspection route at least includes the first imaging device and a second imaging device corresponding to a predetermined position.

[0087] The following is based on Figure 2 the architecture of Figure 4 and the settings of the imaging devices corresponding to the floor plan 400 of Figure 4 shown. Assuming the specified object is a human body for illustration. The processor 110 uses the artificial intelligence model 220 to execute an object detection algorithm on the real-time video signals of the imaging devices 4C1 to 4C5 respectively, and then determines whether the imaging devices 4C1 to 4C5 have captured the specified object. As Figure 4 shown, the artificial intelligence model 220 determines that the imaging device 4C2 and the imaging device 4C5 have captured the users U1 and U2 respectively. Then, the processor 110 further determines the inspection devices in the inspection route according to the relative position information and the imaging device 4C2 and the imaging device 4C5. Here, the processor 110 sets the imaging device 4C2 and the imaging device 4C5 as inspection devices. And since there is no intersection in the shooting ranges of the imaging device 4C2 and the imaging device 4C5, therefore, according to a route setting rule (such as along the aisle of the space or along the production line, or the priority of the imaging devices 4C1 to 4C5) and the relative position information, the processor 110 further selects the imaging device 4C3 and the imaging device 4C4 as inspection devices.

[0088] In addition, in this embodiment, a predetermined position is further set as the starting position or the ending position of the inspection. For example, the "entrance" of the inspection space is set as the predetermined position. Taking Figure 4 as an example, the predetermined position is the door, which corresponds to the imaging device 4C1. The processor 110 further sets the imaging device 4C1 corresponding to the predetermined position as an inspection device. After that, the processor 110 determines the inspection order based on the relative position information as the inspection route of the imaging device 4C1, the imaging device 4C2, the imaging device 4C3, the imaging device 4C4, and the imaging device 4C5 in sequence.

[0089] After setting the inspection order of the imaging devices 4C1 to 4C5, during the inspection process, the display screen can be further controlled by the processor 110 to turn to the direction where the users U1 and U2 are located. For example, taking Figure 4 as an example, the display screen of the display device 130 switches from the real-time video signal of the imaging device 4C1 to the real-time video signal of the imaging device 4C2, then the display screen will turn to the direction of the user U1, and then sequentially switch to the real-time video signals of the imaging device 4C3, the imaging device 4C4, and the imaging device 4C5, and finally the display screen turns to the direction of the user U2.

[0090] In addition, if only one imaging device (such as imaging device 4C1) captures the specified object, in order to achieve the effect of patrol inspection, the processor 110 may use the imaging device 4C2 that captures the specified object and the imaging device 4C1 corresponding to a predetermined position (for example, an entrance corresponding to the space) as patrol inspection devices, and then determine a patrol inspection route including the imaging device 4C1 and the imaging device 4C2 (as patrol inspection devices) according to the relative position information. In addition, in response to only one imaging device (the first imaging device) capturing the specified object, at least three patrol inspection devices may be determined based on the relative position information, the first imaging device that captures the specified object, and the second imaging device and the third imaging device corresponding to two predetermined positions (the position where the patrol inspection starts and the position where the patrol inspection ends).

[0091] In addition, return Figure 1 , taking the target event as an event for instructing patrol inspection of at least one working area, the processor 110 will select the imaging devices corresponding to one or more specified working areas according to the positions where the imaging devices 140-1 to 140-N are distributed as target devices; and determine patrol inspection devices based on the relative position information and the target devices. Here, a predetermined position may be further set as the position where the patrol inspection starts or the position where the patrol inspection ends. The processor 110 determines a plurality of patrol inspection devices based on the relative position information, the target devices, and the imaging devices corresponding to the predetermined positions (the position where the patrol inspection starts or the position where the patrol inspection ends).

[0092] After determining the patrol inspection devices, the processor 110 further determines the patrol inspection order. For example, the patrol inspection order may be determined in ways such as moving clockwise, moving counterclockwise, the smallest rotation angle, and moving along the shortest path. For example, Figures 5A - 5C is a schematic diagram of setting the patrol inspection order according to an embodiment of the present invention. Please refer to Figures 5A - 5C , in this embodiment, assume that there are 4 imaging devices 5C1 to 5C4 that meet the target event. The patrol inspection order of the imaging devices 5C1 to 5C4 may be moving counterclockwise as shown in Figure 5A , or may be moving clockwise as shown in Figure 5B . In addition, as shown in Figure 5C , it may also be set that the patrol inspection order of the imaging devices 5C1 to 5C4 is set in such a way that each imaging device rotates to the next one with the smallest angle. Or, the patrol inspection order of the imaging devices 5C1 to 5C4 may also be set with the shortest action path.

[0093] In addition, in order to connect two imaging devices that meet the target event, an imaging device that does not meet the target event may also be selected as the inspection device. For example, assume that the imaging ranges of two inspection devices (imaging devices that meet the target event) do not overlap. Therefore, when transitioning from the real-time video signal of one inspection device to the real-time video signal of another inspection device, the picture will be discontinuous. Accordingly, in order to connect these two inspection devices, at least one imaging device can be further selected between them as the inspection device.

[0094] Figure 6 is a schematic diagram of setting an inspection route according to an embodiment of the present invention. Please refer to Figure 6 , assume that there are 3 imaging devices 6C1 to 6C3 that meet the target event. Assume that the imaging ranges of imaging device 6C1 and imaging device 6C2 do not overlap, and the imaging ranges between imaging device 6C1 and imaging device 6C3 do not overlap. Accordingly, in addition to setting imaging devices 6C1 to 6C3 as inspection devices, imaging device 6C4 can be further selected between imaging device 6C1 and imaging device 6C2 as the inspection device, and imaging device 6C5 can be selected between imaging device 6C1 and imaging device 6C3 as the inspection device.

[0095] In addition, in the case of multiple target events, the inspection order can also be determined according to the event order of the target events. The processor 110 determines multiple inspection devices included in the inspection route based on multiple target events respectively. Then, based on the event order of these target events and based on the relative position information, the inspection order of the multiple inspection devices is determined. For example, taking the target events including the first event of photographing a human body and the second event of photographing a specified device, and the order of the first event being prior to the order of the second event as an example, the order of the inspection devices that meet the first event is set before the inspection devices that meet the second event.

[0096] After determining the inspection route, in step S315, the processor 110 controls the real-time video signals of each inspection device to be presented on the display device 130 based on the inspection route. That is, the processor 110 switches the display screen of the display device 130 from the real-time video signal of the first inspection device to the real-time video signal of the second inspection device according to the inspection order. Then, the display screen of the display device 130 is switched to the real-time video signal of the third inspection device, and so on, until the display screen of the display device 130 is switched to the real-time video signal of the last inspection device.

[0097] A transition effect can be added to the switching between two real-time video signals to make the display screen visually coherent.

[0098] Figure 7 is a schematic diagram of an inspection movement method according to an embodiment of the present invention. Please refer toFigure 7 In this embodiment, the imaging device 7C1 is described as patrolling toward the imaging device 7C2. And the description is based on the existence of an object T in the direction 72d, but the invention is not limited thereto. The processor 110 controls the imaging device 7C1 to turn from the direction 71d to the direction 72d toward the imaging device 7C2 to capture images, and controls the imaging device 7C2 to face the direction 72d. Then, the processor 110 presents the real-time video signal V1 obtained by the imaging device 7C1 in the direction 72d on the display screen, and controls the imaging device 7C1 to perform a zooming operation to sequentially present the real-time video signals V2 to VN on the display screen, and then switches the display screen to the real-time video signal of the imaging device 7C2. In this way, the display screen is visually coherent.

[0099] In addition, in addition to amplifying the real-time video signal of the imaging device 7C1 to the maximum limit and then transferring it to the real-time video signal of the imaging device 7C2, the inspection method can also move freely within a certain distance (limited range) from the center of the field of view of an imaging device. Figure 7 In the process of jumping from the image capturing device 7C1 to the image capturing device 7C2, the display screen can also move from the limited range of the image capturing device 7C1 to the direction of the image capturing device 7C2. After moving to the limit, the real-time video signal of the image capturing device 7C1 is amplified to the limit through the above-mentioned amplification operation, and then the real-time video signal of the image capturing device 7C2 is switched. This can make the viewer feel more immersed, as if walking in the scene.

[0100] Figure 8 is a schematic diagram of a patrol movement method according to an embodiment of the present invention. Figure 8, in this embodiment, the inspection sequence is the imaging device 8C1, the imaging device 8C2, and the imaging device 8C3 in sequence, and it is illustrated that there is a user U at the imaging device 8C2. The processor 110 controls the imaging device 8C1 to turn to the direction 81d towards the imaging device 8C2 for imaging, and controls the imaging device 8C2 to turn to the direction 81d. Then, the processor 110 presents the real-time video signal of the imaging device 8C1 in the direction 81d to the display screen, and performs a zoom operation on the real-time video signal of the imaging device 8C1 in the display screen. After performing the zoom operation to the maximum limit, the processor 110 controls the imaging device 8C2 to turn from the direction 81d to the direction 82d towards the specified object (i.e., the user U) for imaging, and synchronously switches the display screen to the real-time video signal of the imaging device 8C2 during the turning process of the imaging device 8C2. Accordingly, the display screen can present the visual effect of turning from the direction 81d to 82d. Then, the processor 110 controls the imaging device 8C2 to turn from the direction 82d to the direction 83d towards the imaging device 8C3 for imaging, and presents the real-time video signal acquired by the imaging device 8C2 to the display screen. After that, the processor 110 presents the real-time video signal of the imaging device 8C2 in the direction 83d to the display screen, and performs a zoom operation on the real-time video signal of the imaging device 8C2 in the display screen. After performing the zoom operation to the maximum limit, the display screen is switched to the real-time video signal of the imaging device 8C3.

[0101] During the inspection process, the processor 110 can further use the artificial intelligence model 220 to obtain the real-time information of the specified object, or can also receive the real-time information of the device from the receiving device 230 and further present it to the display screen. For example, on-screen display (OSD), warning lights, pop-up notifications, Internet of Things (IoT), Manufacturing Execution Systems (MES), etc. can be used to present the real-time information.

[0102] In response to the specified object being a device, after the processor 110 detects that there is a device in the real-time video signal by executing the target detection algorithm through the artificial intelligence model 220, the processor 110 obtains the real-time information of the device from the receiving device 230 and records the real-time information. After that, when controlling the display screen to present the real-time video signals of each imaging device, in response to the presence of the specified object (device) in the real-time video signal, when the real-time video signal is presented to the display device 130, the corresponding real-time information is simultaneously presented in the display device 130.

[0103] Figure 9It is a schematic diagram of a display screen according to an embodiment of the present invention. Please refer to Figure 9 , there are three devices (pickling device one, pickling device two, and degreasing tank) in the real-time video signal currently presented on the display screen 900. Accordingly, the processor 110 will simultaneously present three text boxes 910-930 on the display screen 900 to respectively display the real-time information corresponding to the three devices.

[0104] In another embodiment, during the execution of step S315, in response to receiving a selection of a position in the real-time video signal presented in the display device 130, the real-time information corresponding to the specified object or work area included in the position is simultaneously presented in the display device 130. That is, the user can select a position in the real-time video signal presented by the display device 130, and the user can decide the information to be presented at this position, or the processor 110 further identifies whether the selected position in the real-time video signal corresponds to a specified object (such as household appliances, electronic instruments, equipment, building materials) or a work area (such as a test area, a production area, a packaging area). When it is determined that the selected position corresponds to a specified object or a work area, the processor 110 will simultaneously display the real-time information corresponding to the specified object or work area to the display device 130.

[0105] In addition, in response to the specified object being a human body, after the artificial intelligence model 220 detects the presence of a human body in the real-time video signal by executing an object detection algorithm, a selection box for framing the human body is generated by the artificial intelligence model 220. Then, when controlling the display screen to present the real-time video signals of each imaging device, in response to the presence of a specified object (human body) in the real-time video signal, when the real-time video signal is presented to the display device 130, a selection box is simultaneously presented in the display device 130 to frame the human body. In addition, a selection box for framing specific parts such as the palm can also be further generated.

[0106] Figure 10 It is a schematic diagram of a display screen according to an embodiment of the present invention. Please refer to Figure 10 , there is a human body in the video signal currently presented on the display screen 1000. Accordingly, the processor 110 will simultaneously present a selection box 1010 on the display screen 1000 to frame the human body, and further present a selection box 1020 to frame the head of the human body, and present selection boxes 1030 and 1040 to frame the palms of the human body.

[0107] In addition, in response to specifying the object as a human body, after detecting a human body in the real-time video signal through the execution of the target detection algorithm by the artificial intelligence model 222, the artificial intelligence model 222 determines whether the human body is in a dangerous state (for example, falling, not wearing a safety helmet, entering a dangerous area, etc.), and records a warning message when it is determined that the human body is in a dangerous state. After that, when controlling the display screen to present the real-time video signals of the respective imaging devices, in response to the presence of the specified object in the real-time video signal and the specified object having a warning message, when the real-time video signal is presented to the display device 130, the warning message is simultaneously presented in the display device 130. In addition, it can also be set that when the specified object exists in the real-time video signal, when the real-time video signal is presented to the display device 130, the real-time information related to the specified object is simultaneously presented in the display device 130.

[0108] During the process of determining the inspection route and performing inspections among multiple video signals through the display screen, in response to detecting that the new event meets the currently specified target event, the processor 110 further reselects multiple imaging devices as multiple new inspection devices in the imaging device 140. For example, during the inspection process, when it is detected through the artificial intelligence model 220 that another user enters the imaging range of one of the imaging devices, the processor 110 re-executes steps S310 and S315. Based on the relative position information, starting from the imaging device corresponding to the video signal currently displayed on the display device 130 as the inspection starting point, a new inspection route for the new inspection devices is re-determined, and based on the new inspection route, the real-time video signals of the respective new inspection devices are controlled to be presented to the display device 130. That is to say, the inspection system 100 can change the inspection route at any time based on the current situation.

[0109] The processor 110 can further be configured to provide an inspection result interface to the display device 130. Figure 11 It is a schematic diagram of an inspection result interface according to an embodiment of the present invention. Please refer to Figure 11 , the inspection result interface 1100 includes a video block 1110, a floor plan block 1120, an inspection screenshot block 1130, and an information block 1140. The video block 1110 is used to play the real-time video signal in real time. The floor plan block 1120 is used to display the floor plan corresponding to the space where the imaging device is located, and the floor plan includes the position information corresponding to the actual positions of the respective imaging devices set in the space in the floor plan and the trajectory based on the inspection order. The inspection screenshot block 1130 is used to display the screenshots corresponding to the target events, for example, display the screenshots of each device. The information block 1140 is used to display the real-time information corresponding to the target events. For example, Figure 11The embodiments specify two target events, namely the event of photographing a human body and the event of photographing a device. Therefore, the real-time information of each device, such as "Device #03 repaired" and "Device #06 started", will be synchronously displayed in the information block 1140, and when it is determined that a human body is in a dangerous state, the corresponding warning information, such as "Area A: Personnel without safety helmets", will be displayed.

[0110] In another embodiment, the real-time information and / or warning information can also be directly superimposed on the real-time video signal presented on the display screen. Figure 12 It is a schematic diagram of a display screen according to an embodiment of the present invention. Please refer to Figure 12 , in the display screen 1200, the selection boxes 12F1 - 12F5, the selection boxes 12W1 - 12W2, and the text boxes 1210 - 1250 are presented simultaneously. The selection boxes 12F1 - 12F5 are used to frame the human body in the real-time video signal. The selection boxes 12W1 - 12W2 are used to frame the specified building materials in the real-time video signal. The text box 1210 is used to present the number of people detected in the currently displayed real-time video signal. The text box 1220 corresponds to the selection box 12F2 and is used to present the warning signal of the human body framed by the selection box 12F2, such as "Not wearing a safety helmet". The text box 1230 is used to present the real-time information of the detected device (such as a pickling tank), such as the operating conditions of temperature, concentration, etc. The text boxes 1240 and 1250 correspond to the selection boxes 12W1 - 12W2 respectively and are used to present the real-time information of the building materials framed by the selection boxes 12W1 - 12W2, such as the operation content and production capacity status that this building material should carry out.

[0111] Figure 13 It is a schematic diagram of a display screen according to an embodiment of the present invention. Please refer to Figure 13 , in the display screen 1300, the selection boxes 13W1 - 13W3 are presented simultaneously. The selection box 13W1 and the selection box 13W2 frame the iron hook, and the selection box 13W3 frames the building material. The processor 110 can further detect the angle between the lifted building material and the horizontal plane and synchronously present the angle information in the display screen 1300, and dynamically change the presented angle information as the actual operating angle changes.

[0112] Figure 14 It is a schematic diagram of an inspection route according to an embodiment of the present invention. Please refer to Figure 14, imaging devices 14C1 to 14C4 are provided in the space of this embodiment, and users 14U1 to 14U4 exist in this space. Since the imaging device 14C2 does not capture a person, the imaging device 14C2 is not set as a patrol device. After the processor 110 determines the patrol devices as the imaging devices 14C1, 14C3, 14C4 and the patrol order, first, the processor 110 controls the display screen of the display device 130 to present the real-time video signal of the imaging device 14C1, then controls the display screen to turn to the direction 14d1 of the user 14U1, and then controls the display screen to display the real-time video signal of the imaging device 14C1 turning towards the direction 14d2 of the imaging device 14C3.

[0113] After that, the processor 110 controls the display screen of the display device 130 to switch to the real-time video signal of the imaging device 14C3 (towards the direction 14d2), then controls the display screen to turn to the direction 14d3 of the user 14U2, and then to the direction 14d4 towards the imaging device 14C4. Then, the processor 110 controls the display screen of the display device 130 to switch to the real-time video signal of the imaging device 14C4 (towards the direction 14d4), then controls the display screen to turn to the direction 14d5 of the user 14U3, and then to the direction 14d6 of the user 14U4.

[0114] Figure 15 It is a schematic diagram of a patrol route according to an embodiment of the present invention. Please refer to Figure 15 , devices 1510, 1520 and imaging devices 15C1, 15C2 are provided in the space of this embodiment, and users 14U1 to 14U4 exist in this space. The processor 110 determines the patrol order of the imaging devices 15C1, 15C2 as the imaging device 15C1 going to the imaging device 15C2, and sequentially executes the following steps A to D. In step A, the processor 110 controls the imaging device 15C1 to image the device 1510 to present the acquired real-time video signal on the display screen of the display device 130. After that, in step B, the processor 110 controls the imaging device 15C1 to image towards the direction 15d1 of the imaging device 15C2, and controls the imaging device 15C2 to also image towards the direction 15d1, so that the display screen switches from the real-time video signal of the imaging device 15C1 to the real-time video signal of the imaging device 15C2 (towards the direction 15d1).

[0115] Next, in step C, the processor 110 controls the imaging device 15C2 to image the device 1520, so as to present the acquired real-time video signal on the display screen of the display device 130. Finally, in step D, the processor 110 controls the imaging device 15C2 to image in the direction 15d2 of the imaging device 15C1, and controls the imaging device 15C1 to also image in the direction 15d2, so that the display screen switches from the real-time video signal of the imaging device 15C2 to the real-time video signal of the imaging device 15C1 (towards the direction 15d2). Then, steps A to D are repeatedly executed. Since the target events of this embodiment are to capture the specified devices 1510 and 1520, the imaging devices 15C1 and 15C2 will not specifically turn to the direction where the user is located. In Figure 15 In the illustrated embodiment, for example, when the imaging device 15C1 turns from the direction 15d2 to the device 1510, the user 15U2 will be captured and displayed on the display screen.

[0116] In addition, for users who do not appear in the patrol route, a picture-in-picture (PIP) can be used to present the users who do not appear in the patrol route. For example, Figure 15 in terms of, after the display screen in step D switches to the real-time video signal of the imaging device 15C1 (towards the direction 15d2), the picture-in-picture can be used to display the users 15U1, 15U2, and 15U3.

[0117] In summary, the present invention provides a patrol method and a patrol system, which can select a device that meets the target event from multiple imaging devices, generate a patrol route based on this, and then concentrate and display the content acquired by the imaging device based on the patrol route. Accordingly, the picture that meets the target event can be quickly acquired from multiple real-time video signals and displayed on the display device.

Claims

1. A patrol inspection method, suitable for being executed by an electronic device, characterized in that, The inspection method includes: Obtaining relative position information between multiple imaging devices; Determining an inspection route based on a target event and the relative position information, where the inspection route meets the target event, and multiple imaging devices passed by in the inspection route are set as multiple inspection devices; and Controlling real-time video signals of the multiple inspection devices to be presented on a display device based on the inspection route.

2. The inspection method according to claim 1, wherein It further includes: Establishing the relative position information between the multiple imaging devices, including: Providing a floor plan corresponding to the space where the multiple imaging devices are arranged; Based on user operations, marking multiple planar positions corresponding to the actual positions of the multiple imaging devices arranged in the space in the floor plan; and Calculating the relative position information between the multiple imaging devices based on the multiple planar positions.

3. The inspection method according to claim 1, wherein It further includes: Establishing the relative position information between the multiple imaging devices, including: Respectively obtaining multiple images corresponding to the multiple imaging devices from the multiple imaging devices; And Calculating the relative position information between the multiple imaging devices by finding corresponding feature points in every two images.

4. The inspection method according to claim 1, characterized in that The target event includes an event for indicating that a specified object is photographed. The step of determining the inspection route includes: Judging whether the multiple imaging devices photograph the specified object by executing a target detection algorithm on the real-time video signals received by each of the multiple imaging devices through an artificial intelligence model; and In response to multiple target devices among the multiple imaging devices photographing the specified object, determining the multiple inspection devices based on the relative position information and the multiple target devices that photograph the specified object, where the number of the multiple inspection devices included in the inspection route is greater than or equal to the number of the multiple target devices.

5. The inspection method according to claim 4, wherein after judging whether the multiple imaging devices photograph the specified object, it further includes: In response to only the first imaging device among the multiple imaging devices photographing the specified object, determining the inspection route based on the relative position information and the first imaging device that photographs the specified object, where the inspection route at least includes the first imaging device and a second imaging device corresponding to a predetermined position.

6. The inspection method according to claim 4, wherein after judging whether the multiple imaging devices photograph the specified object, it further includes: In response to the specified object being a device, after detecting the device in the real-time video signal by executing the target detection algorithm through the artificial intelligence model, obtaining real-time information of the device and recording the real-time information; Wherein the step of controlling real-time video signals of the multiple inspection devices to be presented on the display device further includes: In response to the specified object existing in the real-time video signal, when the real-time video signal is presented on the display device, presenting the corresponding real-time information on the display device at the same time.

7. The inspection method according to claim 4, wherein after determining whether the plurality of imaging devices have captured the specified object, it further includes: In response to the specified object being a human body, after detecting the presence of the human body in the real-time video signal by executing the target detection algorithm through the artificial intelligence model, determining whether the human body is in a dangerous state through the artificial intelligence model, and recording a warning message when it is determined that the human body is in the dangerous state; Wherein the step of controlling the real-time video signals of the plurality of inspection devices to be presented to the display device further includes: In response to the presence of the specified object in the real-time video signal and the specified object having the warning message, when the real-time video signal is presented to the display device, presenting the warning message in the display device simultaneously.

8. The inspection method according to claim 4, wherein after determining whether the plurality of imaging devices have captured the specified object, it further includes: In response to the specified object being a human body, after detecting the presence of the human body in the real-time video signal by executing the target detection algorithm through the artificial intelligence model, generating a selection box that frames the human body through the artificial intelligence model; Wherein the step of controlling the real-time video signals of the plurality of inspection devices to be presented to the display device further includes: In response to the presence of the specified object in the real-time video signal, when the real-time video signal is presented to the display device, presenting the selection box in the display device to frame the human body simultaneously.

9. The inspection method according to claim 4, wherein the step of controlling the real-time video signals of the plurality of inspection devices to be presented to the display device based on the inspection route includes: Switching the display screen of the display device from the real-time video signal of a first inspection device among the plurality of inspection devices to the real-time video signal of a second inspection device that has captured the specified object among the plurality of inspection devices, including: Controlling the first inspection device to turn to image in a first direction towards the second inspection device, and controlling the second inspection device to turn to the first direction; Presenting the real-time video signal of the first inspection device in the first direction to the display screen; Performing a magnification operation on the real-time video signal of the first inspection device in the display screen; and After performing the magnification operation, controlling the second inspection device to turn from the first direction towards a second direction of the specified object to image, and synchronously switching the display screen to the real-time video signal of the second inspection device during the turning process of the second inspection device.

10. The inspection method according to claim 1, wherein the target event includes an event for instructing the inspection of at least one working area, The step of determining the inspection route includes: Selecting at least one target device corresponding to the at least one working area among the plurality of imaging devices; And Determining the plurality of inspection devices based on the relative position information and the at least one target device.

11. The inspection method according to claim 1, further includes: Determine the plurality of inspection devices included in the inspection route based on the target event and at least one other target event; and Determine the inspection order of the plurality of inspection devices with reference to the event order of the target event and at least one other target event, and based on the relative position information.

12. The inspection method according to claim 1, wherein the step of determining the inspection route includes: Determine the inspection order of the plurality of inspection devices based on the relative position information and the priorities of the plurality of imaging devices.

13. The inspection method according to claim 1, wherein in the process of sequentially displaying the video signals of each of the plurality of inspection devices to the display device based on the inspection route, it further includes: In response to detecting that a new event meets the target event, reselect a plurality of the plurality of imaging devices as a plurality of new inspection devices; Based on the relative position information, with the imaging device corresponding to the video signal currently displayed on the display device as the inspection starting point, re-determine a new inspection route for the plurality of new inspection devices; and Based on the new inspection route, control the real-time video signals of the plurality of new inspection devices to be presented to the display device.

14. The inspection method according to claim 1, further includes: Provide an inspection result interface to the display device, where the inspection result interface includes a video block, a floor plan block, an inspection screenshot block, and an information block, The video block is used to play the real-time video signal in real time, The floor plan block is used to display the floor plan corresponding to the space where the plurality of imaging devices are located, and the floor plan includes a plurality of position information corresponding to the actual positions of the plurality of imaging devices arranged in the space in the floor plan and a trajectory based on the inspection order, The inspection screenshot block is used to display the screenshot corresponding to the target event, The information block is used to display the real-time information corresponding to the target event.

15. The inspection method according to claim 1, wherein in the process of controlling the real-time video signals of the plurality of inspection devices to be presented to the display device based on the inspection route, it further includes: In response to receiving a selected position in the real-time video signal presented on the display device, simultaneously present the real-time information corresponding to the specified object or work area included in the position on the display device.

16. An inspection system, characterized in that, Includes: A plurality of imaging devices; A display device; and A processor, coupled to the plurality of imaging devices and the display device, wherein the processor is configured to: Obtain the relative position information between the plurality of imaging devices; Determine an inspection route based on a target event and the relative position information, where the inspection route meets the target event, and multiple imaging devices passing through in the inspection route are set as a plurality of inspection devices; and Based on the inspection route, control the real-time video signals of the plurality of inspection devices to be presented to the display device.

Images