Image judgment device and image judgment system
By designing an image judgment device that can receive and process a variety of image data, the problem that the existing system can only process the output data of IP cameras is solved, and the judgment of a variety of image data is realized, and the applicability of the system is improved.
Patent Information
- Application Number
- CN202010751845.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2020-06-18
- Filing Date
- 2020-07-30
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2040-07-30
AI Technical Summary
The existing image determination system can only process image data output by the IP camera, but cannot process image data from other than the camera device.
An image judging device is designed, including a receiving unit, a playback unit, a designating unit and a judgment unit, which can receive image data from the imaging device and other devices, and play and detect area specification on the display device to realize the use of a variety of images. Image judgment of data.
The object of image judgment is expanded, not only limited to image data output by the imaging device, but also process image data output from devices other than the imaging device, improving the flexibility and applicability of the system.
Smart Images

Figure CN112307882B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to an image determination device and an image determination system. Background Art
[0002] Image recognition systems are used to analyze recorded image data using a camera or other imaging device to detect a target object or to judge the movement or abnormality of an object in the image data. Image recognition systems have been significantly developed in recent years. Such image recognition systems are applied to, for example, product production lines.
[0003] As an image monitoring system, the image is used to judge the movement or abnormality of an object based on an image. The prior art, such as patent document 1 (JP Patent Gazette No. 6459256), discloses the following technical solution, namely, it has an image extraction unit for extracting an image from an image of a monitored object captured with the monitored object according to the working area or working time conditions of the operator set by the user; a list generation unit for associating a tag associated with a specific event set by the user for the working area, working time or working content with the image extracted by the image extraction unit, generating a list, and saving it in a storage device; and a playback control unit for reading the list saved in the storage device and displaying it on the display device. At the same time, when a tag is selected from the list displayed on the display device, the image associated with the tag is found and played and displayed on the display device.
[0004] However, the technical solution disclosed in Patent Document 1 has the following problem, that is, the device connected to the image judgment device can only be a camera device such as an IP (Internet Protocol) camera (network camera), and can only judge and process the image data taken by the camera device. Summary of the invention
[0005] In view of the above-mentioned problems, the present invention provides an image judgment device and an image judgment system, whose purpose is to not only limit the image data that can be the judgment object to the image data output by the camera device, but also allow image data from devices other than the camera device to be used as the judgment object.
[0006] In order to achieve the above-mentioned purpose, the present invention provides an image judgment device that can be used to perform image judgment on image data, comprising: a receiving unit for receiving at least the second image data among first image data and second image data, the first image data being image data output from a camera device, and the second image data being image data output from another device, which is different from the photographic device that outputs the image data that becomes the image judgment object; a first playback unit for playing and displaying at least the second image data received by the receiving unit on a display device; a designation unit for designating a detection area that becomes the image judgment object in a first display area of the display device that plays and displays the first image data or the second image data; and a judgment unit for performing the image judgment on the first image data or the second image data within the detection area of the second display area of the display device that displays at least one of the first image data and the second image data.
[0007] The present invention has the effect that the image data to be determined is not limited to the image data output by the imaging device, and image data from a device other than the imaging device can also be used as the determination target. BRIEF DESCRIPTION OF THE DRAWINGS
[0008] Figure 1 It is a schematic diagram of the overall structure of the image determination system involved in the embodiment.
[0009] Figure 2 It is a schematic diagram of the hardware configuration of the information processing device involved in the embodiment.
[0010] Figure 3 It is a functional block diagram of the information processing device involved in the embodiment.
[0011] Figure 4 It is a schematic diagram of the directory structure of information stored in the storage device of the information processing device involved in the implementation method.
[0012] Figure 5 This is a schematic diagram showing a plurality of real-time image data on an observation screen displayed on a display device of the information processing device according to the embodiment.
[0013] Figure 6 This is a diagram showing an example of a state in which image data is displayed on a setting screen displayed on the display device of the information processing device according to the embodiment.
[0014] Figure 7 This is a schematic diagram for designating a detection area in a setting screen display unit on a setting screen displayed on a display device of the information processing apparatus according to the embodiment.
[0015] Figure 8This is a schematic diagram of a state in which a monitoring operation is performed on an observation screen displayed on a display device of the information processing device according to the embodiment.
[0016] Fig. 9 This is a schematic diagram of image determination in a detection area in a monitoring operation performed on an observation screen displayed on a display device of the information processing device according to the embodiment.
[0017] Fig.10 This is a schematic diagram of the overall structure of the image determination system according to the embodiment, in which a virtual camera is used as the image determination system.
[0018] Fig.11 This is a schematic diagram of the state of the observation screen output from the image determination system as a virtual camera on the observation screen displayed on the display device of the information processing device according to the embodiment.
[0019] Fig.12 This is a schematic diagram of a state where the image determination system according to the embodiment is connected in multiple stages as a virtual camera.
[0020] Fig.13 This is a schematic diagram showing a state where recorded image data is played and displayed on a browser screen displayed by a display device of the information processing device according to the embodiment.
[0021] Fig.14 This is a schematic diagram showing a state where a mark list screen is displayed on a browser screen displayed by a display device of the information processing device according to the embodiment.
[0022] Fig.15 2 is a schematic diagram showing the overall configuration of an image determination system according to Modification 1.
[0023] Fig.16 It is a schematic diagram of the overall structure of the image determination system of Modification Example 2.
[0024] Fig.17 This is a schematic diagram for explaining an image of software that performs a sound visualization process as an image output by a virtual camera connected to the image determination system of Modification 3.
[0025] Fig.18 It is a schematic diagram of the overall structure of the image determination system involved in Modification Example 4.
[0026] Fig.19 This is a schematic diagram for explaining an image output by a virtual camera as an image judgment system connected to deformation example 5, and an image of software for making a chart using data dot.
[0027] Fig. 20 It is a schematic diagram of the overall structure of the image determination system of Modification Example 6. DETAILED DESCRIPTION
[0028] The following is a detailed description of the embodiments of the image recognition device and the image recognition system according to the present invention with reference to the accompanying drawings. The present invention is not limited by the embodiments, and the constituent elements of the following embodiments include those that are easily conceivable by a person skilled in the art of the present invention, are substantially the same, and are so-called equivalent. The constituent elements may be omitted, replaced, changed, and combined within the scope of the following embodiments without departing from the spirit of the following embodiments.
[0029] [Overall structure of image processing system]
[0030] Figure 1 FIG. 1 is a schematic diagram of an overall structure of an image recognition system according to an embodiment of the present invention. Figure 1 , the overall structure of the image determination system 1 involved in this embodiment is described.
[0031] like Figure 1 As shown, the image determination system 1 includes imaging devices 2 b to 2 f , a virtual camera 5 , a hub 4 , and an information processing device 3 .
[0032] The camera devices 2b to 2f are IP cameras (network cameras) that convert light from the subject into an electrical signal to photograph (photograph) the subject, generating image data (first image data) of an animation (e.g., 10 [FPS]) consisting of a plurality of frames. For example, the camera devices 2b to 2f photograph a production facility or production line for manufacturing a product, and generate image data for detecting abnormalities in a workpiece for manufacturing a product or determining errors in an operator's operating procedure.
[0033] When the imaging devices 2b to 2f are not distinguished or are collectively referred to, they are simply referred to as "imaging device 2". Figure 1 The image determination system 1 includes five imaging devices 2 , but the present invention is not limited thereto, and may include another number of imaging devices 2 .
[0034] The virtual camera 5 is not a device that directly outputs the photographic image or the like as a judgment object to the information processing device 3 like the photographic devices 2b to 2f, but is a device or system that can output image data (an example of second image data) such as animation, photographic images, and generated images as judgment objects for abnormality or the like. In this embodiment, the virtual camera 5 is another image judgment system having a configuration equivalent to the image judgment system 1 as described below. The virtual camera 5 is equivalent to the "another device" of the present invention.
[0035] exist Figure 1In the embodiment, the image determination system 1 is configured to include one virtual camera 5, but the present invention is not limited thereto and may also include a configuration including a different number of virtual cameras 5. The image data is described as a video image composed of a plurality of frames, but includes a still image composed of one frame. Here, the image data output from the camera devices 2b to 2f is referred to as the first image data, and the image data output from the virtual camera 5 is referred to as the second image data, but this is only used to distinguish whether the output source of the image data is the camera devices 2b to 2f or the virtual camera 5.
[0036] The hub 4 is a switching hub that corresponds to, for example, Ethernet (registered trademark) standards and is used to connect the camera devices 2b to 2f, the virtual camera 5, and the information processing device 3 to form a LAN (Local Area Network). The camera devices 2b to 2f, the virtual camera 5, and the information processing device 3 each have an IP address (and a MAC (Media Access Control) address), so a LAN can be formed by the hub 4 as a switching hub. In this case, the LAN performs data communication using protocols such as TCP (Transmission Control Protocol) / IP (Internet Protocol).
[0037] The information processing device 3 is a PC (Personal Computer) or a workstation etc. having an image judgment device function for performing image judgment based on the image data (first image data) output from the camera device 2 and the image data (second image data) such as animation output from the virtual camera 5. The information processing device 3 is connected to an external device 10 such as a production device via a field bus, for example, so as to enable data communication. The communication standards of the field bus include, for example, Profibus used in Europe or CC-Link (registered trademark) of Mitsubishi Electric Corporation.
[0038] exist Figure 1 The LAN shown shows an example of TCP / IP communication, but the present invention is not limited to this. For example, the information processing device 3 may have multiple VGA (Video Graphics Array) terminals or USB (Universal Serial Bus) ports, and multiple camera devices 2 or virtual cameras 5 may be connected to the information processing device 3 via VGA cables or USB cables.
[0039] [Hardware Configuration of Information Processing Device]
[0040] Figure 2 FIG. 1 is a schematic diagram showing an example of the hardware configuration of an information processing device according to an embodiment of the present invention. Figure 2The hardware configuration of the information processing device 3 according to the present embodiment will be described.
[0041] like Figure 2 As shown, the information processing device 3 includes a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, an auxiliary storage device 104, a display 105 (display device), a network I / F 106, a keyboard 107 (input device), a mouse 108 (input device), a DVD (Digital Versatile Disc) drive 109, a peripheral I / F 111, and a speaker 112.
[0042] The CPU 101 controls the operation of the entire information processing device 3. The ROM 102 is a nonvolatile memory that stores programs for the information processing device 3 and BIOS (Basic Input / Output System). The RAM 103 is a volatile memory used as a work area for the CPU 101.
[0043] The auxiliary memory 104 is a storage device such as a hard disk (Hard Disk Drive) or SSD (Solid State Drive) for storing image data (first image data) output by the camera 2, image data such as animation output by the virtual camera 5 (second image data), and various data such as setting information.
[0044] The display 105 is used to display various information such as a cursor, menu, window, text or image, or a screen of an application program that performs an image judgment operation of the information processing device 3. The display 105 is, for example, a CRT (Cathode Ray Tube) display, a liquid crystal display (LCD) or an organic EL (Electroluminescence) display. The display 105 is connected to the host of the information processing device 3 via a VGA cable or an HDMI (registered trademark) (High-Definition Multimedia Interface) cable, or an Ethernet cable.
[0045] The network I / F 106 is an interface for connecting to the hub 4 for data communication. The network I / F 106 is a NIC (Network Interface Card) that can communicate using, for example, the TCP / IP protocol. Specifically, the information processing device 3 obtains image data from the camera 2 and the virtual camera 5 via the hub 4 and the network I / F 106. The network I / F 106 may also be a wireless I / F that complies with communication standards such as Wi-Fi (registered trademark).
[0046] The keyboard 107 is an input device for selecting characters, numbers, various instructions, moving a cursor, setting setting information, etc. The mouse 108 is an input device for selecting and executing various instructions, selecting a processing object, moving a cursor, and setting setting information, etc.
[0047] The DVD drive 109 is a device for controlling the reading, writing, and deletion of data from a DVD 110, which is an example of a removable storage medium. The storage medium read or written by the DVD drive 109 is not limited to a DVD, but may also be a CD-ROM (Compact Disc-Read Only Memory), CD-R (Compact Disc-Recordable), DVD-R, DVD-RW (DVDReWritable), or a Blu-ray disc.
[0048] The peripheral I / F 111 is an interface for connecting to the external device 10 for data communication. The external device I / F 111 is an interface for communication via, for example, a field bus. Specifically, the information processing device 3 communicates data with the external device 10 via the peripheral I / F 111.
[0049] The speaker 112 is a device that outputs sound according to the operation of the application program.
[0050] The CPU 101, ROM 102, RAM 103, auxiliary storage device 104, display 105, network I / F 106, keyboard 107, mouse 108, DVD drive 109, peripheral I / F 111, and speaker 112 communicate with each other via a bus 113 such as an address bus and a data bus. When the display 105 is connected via an Ethernet cable, it is connected to the network I / F 106, and in this case, data communication is performed according to a protocol such as TCP / IP.
[0051] [Module structure of information processing device]
[0052] Figure 3 FIG. 1 is a schematic diagram of an example of a functional module structure of an information processing device according to an embodiment of the present invention. Figure 3 , describing the functional module structure of the information processing device 3 involved in this embodiment.
[0053] like Figure 3 As shown, the information processing device 3 includes an image receiving unit 301 (receiving unit), a storage unit 302, an input unit 303 (input device), a first playback unit 304, a setting unit 308, a detection area specifying unit 309 (specifying unit), an image assigning unit 312, a detection area determining unit 314 (determining unit), a first control unit 316, a second playback unit 318, an area display control unit 319, a list control unit 320, a second control unit 321, a display control unit 331, an external output unit 332, and a display unit 333 (display device). To simplify the description, Figure 3 The hub 4 is omitted in the figure.
[0054] The image receiving unit 301 is used to perform data communication with the camera device 2 and the virtual camera 5, and receives image data from the camera device 2 and image data from the virtual camera 5. The image receiving unit 301 stores the received image data in the storage unit 302. Figure 2 The network I / F 106 and the CPU 101 shown are implemented by executing a program.
[0055] The storage unit 302 is a storage device for storing image data, various setting information, and time information described later, etc. received by the image receiving unit 301. The storage unit 302 identifies the image data received by the image receiving unit 301 and the image data or image generated by which camera 2 or virtual camera 5, and then stores the image data. The storage unit 302 is composed of, for example, Figure 2 The memory is implemented by the RAM 103 or the auxiliary storage device 104 shown.
[0056] The input unit 303 is used to perform operation input for image determination operation etc. performed by the information processing device 3. Figure 2 The keyboard 107 and mouse 108 shown are implemented.
[0057] The first playback unit 304 sends the image data obtained from the storage unit 302 to the display control unit 331 according to the operation signal of the input unit 303 operated by the user, so that the display control unit 331 plays and displays the image data on the display unit 333. Figure 6 The first playback unit 304 plays and displays the image data on the setting image display unit 502 of the setting screen 500 displayed on the display unit 333 described later. Figure 2 This is achieved by the CPU 101 shown running a program.
[0058] The setting unit 308 is used to set various setting information and store it in the storage unit 302 according to the operation signal from the input unit 303 operated by the user. The setting unit 308 sends the information to be displayed from the storage unit 302 to the display control unit 331, and the display control unit 331 displays the setting information on the display unit 333. The setting unit 308 is Figure 2 This is achieved by the CPU 101 shown running a program.
[0059] The detection area designation unit 309 is used to designate a detection area as an object of image judgment action such as detecting abnormalities in the display area of the image data of the display unit 333 (the setting image display unit 502) (the first display area) in accordance with an operation signal from the input unit 303 that is operated by the user. The detection area designation unit 309 saves the information of the designated detection area in the storage unit 302 in association with the camera device 2 or the virtual camera 5. The information of the detection area refers to information such as the position coordinates of the detection area in the image data or the image display area, and the shape of the detection area. The detection area designated by the detection area designation unit 309 is used in the detection function described later. The detection area designation unit 309 is stored in the storage unit 302 by associating the detection area with the camera device 2 or the virtual camera 5. Figure 2 This is achieved by the CPU 101 shown executing a program.
[0060] In addition to the above-mentioned abnormality judgment (abnormality detection), the above-mentioned image judgment action also envisions whether a specific phenomenon occurs, etc. However, in order to simplify the explanation, this embodiment uses the abnormality judgment (abnormality detection) action as an example of the image judgment action.
[0061] The image distribution unit 312 is used to send the real-time image data obtained from the image receiving unit 301 to the display control unit 331 according to the operation signal from the input unit 303 operated by the user, so that the display control unit 331 displays the image data and the image on the display unit 333. Specifically, the image distribution unit 312 distributes the corresponding image data and the image to Figure 5 The image distribution unit 312 stores the image data displayed on the image display units 401a to 401f in the storage unit 302 (recording). Figure 2 The CPU 101 shown executes a program to achieve this.
[0062] The detection area determination unit 314 is used to monitor the operation after it starts. Figure 5 In the detection area of the image display unit 401a to 401f of the observation screen 400, the detection reference image is compared with the image of the detection area in the frame or image as the image data to determine whether there is an abnormality. The detection area determination unit 314 is Figure 2This is achieved by the CPU 101 shown executing a program.
[0063] The first control unit 316 is used to control the input unit 303 according to the operation signal received from the user, except for the following Figure 5 The first control unit 316 is configured to start and stop the monitoring operation on the observation screen 400, and also to control the operation of the observation screen 400 as a whole. Figure 2 This is achieved by the CPU 101 shown running a program.
[0064] The second playback unit 318 is used to send the image data obtained from the storage unit 302 to the display control unit 331 according to the operation signal from the input unit 303 operated by the user, so that the display control unit 331 can display the image data and the image on the display unit 333. Specifically, the second playback unit 318 allocates the corresponding image data to Fig.13 The second playback unit 318 is used to play and display the browser screen 700 displayed on the display unit 333 described later. Figure 2 This is achieved by the CPU 101 shown running a program.
[0065] The area display control unit 319 is used to overlap and display the status of each detection area on the frame or image of the image data played and displayed in the image display units 701a to 701f of the browser screen 700, based on the information (time information) of the detection time of each detection area stored in the input unit 303. The area display control unit 319 is Figure 2 This is achieved by the CPU 101 shown in FIG. 1 running a program.
[0066] The list control unit 320 is used to read the markup file stored in the storage unit 302 in the browser screen 700, and display the content of the markup file. Fig.14 The mark list screen 800 described later is displayed on the screen. Here, the mark file is a file that records the time when the abnormality is detected by the detection function in time series. The list control unit 320 is Figure 2 This is achieved by the CPU 101 shown in FIG. 1 running a program.
[0067] The second control unit 321 is used to control the overall operation of the browser screen 700. Figure 2 This is achieved by the CPU 101 shown running a program.
[0068] The display control unit 331 is used to control the display operation of the display unit 333. Specifically, the display control unit 331 displays the image data obtained from the first playback unit 304 and the image distribution unit 312, the setting information set by the setting unit 308, and the detection area specified by the detection area specifying unit 309 on the display unit 333. The display control unit 331 is Figure 2 This is achieved by the CPU 101 shown running a program (driver program).
[0069] The external output unit 332 is used to output an abnormality signal to the external device 10 when the detection area determination unit 314 makes an abnormality determination. Figure 2 The external output unit 332 outputs an abnormality signal, but the present invention is not limited thereto, and measures such as sending a batch file for notifying an abnormality by email to the external device 10 or notifying an abnormality on the observation screen 400 (for example, by flashing a light icon) may also be taken.
[0070] The display unit 333 is a device that displays various data according to the control of the display control unit 331. The display unit 333 is particularly configured by Figure 2 The CPU 101 shown runs a program (application) to display an observation screen 400, a setting screen 500, and a browser screen 700, etc., which will be described later. The display unit 333 is a display unit that is configured to display the image of the user. Figure 2 The display 105 shown is implemented.
[0071] Figure 3 Part or all of the first playback unit 304, setting unit 308, detection area specifying unit 309, image allocation unit 312, detection area determining unit 314, first control unit 316, second playback unit 318, area display control unit 319, list control unit 320, second control unit 321 and display control unit 331 shown can be implemented using hardware circuits such as ASIC (Application Specific Integrated Circuit) or FPGA (Field-Programmable Gate Array) instead of software programs.
[0072] Figure 3 The functional units shown are conceptually represented, but the present invention is not limited to such a configuration. Figure 3 The independent functional units in the figure can also be composed of multiple functional units as one functional unit. Figure 3 A functional part of can also be divided into multiple functions and constituted as multiple functions.
[0073] In order to simplify the following description, displaying the data received by the display control unit 331 on the display unit 333 is simply described as sending the data to the functional unit of the display control unit 331 so that the display unit 333 (or the screen displayed by the display unit 333) displays the data. For example, when the first playback unit 304 sends image data to the display control unit 331 and the display control unit 331 displays the image data on the display unit 333, it is simply described as the first playback unit 304 plays and displays the image data on the display unit 333.
[0074] [Directory structure of information stored in the storage unit]
[0075] Figure 4 FIG. 1 is a schematic diagram of an example directory structure of information stored in a storage device in an information processing device according to an embodiment. Figure 4 , describing the directory structure used to manage information in the storage unit 302.
[0076] First, if Figure 4 As shown, the storage unit 302 forms a "Setting" folder and a "Common folder" as root folders. The "Setting" folder forms "CAM1" to "CAM6" folders (hereinafter referred to as "camera folders") for storing information about each camera 2 and the virtual camera 5. The "CAM1" to "CAM6" folders correspond to the folders of the virtual camera 5 and the cameras 2b to 2f, respectively.
[0077] like Figure 4 As shown, each camera folder stores a region information file, a time information file, and image data. Here, the region information file contains the position information (information such as coordinates indicating the position and the shape of the region) of each detection region set for the corresponding camera 2 or virtual camera 5, and the attribute information described later. The time information file contains the time when the monitoring action for the corresponding camera 2 or virtual camera 5 starts and ends, the time when each detection area is designated (set), and the time when each detection area is detected during the monitoring action. In addition, the image data is data such as the detection reference image described later.
[0078] In addition, the above-mentioned tag files are saved in the "Common" folder.
[0079] like Figure 4 The directory structure of the management information shown is an example, and the present invention is not limited thereto, and information can also be managed by other data structures. Figure 4 The folder names shown are also examples and are not limited thereto.
[0080] [Observation screen composition]
[0081] Figure 5 FIG. 1 is a schematic diagram showing an example of a plurality of real-time image data being displayed on an observation screen displayed on a display device of an information processing device according to an embodiment. Figure 5 , explaining the composition of the observation screen 400 and the actions represented by the plurality of real-time image data.
[0082] The program (application) executed by the CPU 101, Figure 5 The observation screen 400 is displayed on the display 333. The observation screen 400 is a screen that displays the image data received from the plurality of imaging devices 2 and the image data received from the virtual camera 5 in real time, and performs image judgment in each designated detection field when performing an image judgment action (monitoring action). Specifically, in the observation screen 400, when performing an image judgment action (monitoring action), the detection field judgment unit 314 judges whether there is an abnormality in the detection field designated by the detection field designation unit 309.
[0083] Observe screen 400 as Figure 5 As shown, it includes image display units 401a to 401f, a monitoring start button 411, a monitoring stop button 412, a monitoring setting button 413, an OK light 414, an NG light 415, and status display units 421a to 421f.
[0084] The image display units 401a to 401f are display areas that respectively display real-time image data obtained from the camera devices 2a to 2f by the image distribution unit 312 via the image receiving unit 301 in accordance with an operation signal from the input unit 303 operated by the user. The image display units 401a to 401f can be selected when pressed by the user operating the input unit 303 (for example, a click operation of the mouse 108). When the image display units 401a to 401f are not distinguished or collectively referred to, they are simply referred to as "image display unit 401" (second display area).
[0085] The monitoring start button 411 is a button for starting the determination of the presence or absence of abnormality in the detection area designated by the detection area designation unit 309 in the image display unit 401 displaying the selected state of real-time image data when pressed by the user's operation of the input unit 303 .
[0086] The monitoring stop button 412 is a button for stopping the operation of determining whether or not an abnormality exists in the detection area by the detection area determination unit 314 in the image display unit 401 in the selected state when pressed by the user's operation on the input unit 303 .
[0087] When the monitoring setting button 413 is pressed by the user's operation on the input unit 303, the first control unit 316 displays a display 333 to be described later. Figure 6The setting screen 500 shown in FIG. 1 is used to set buttons for each determination area.
[0088] The OK light 414 is a light area that is lit by the first control unit 316 when the result of the determination of whether there is an abnormality in the designated detection area is that there is no abnormality in each image display unit 401 in the selected state. For example, when the OK light 414 is in the selected image display unit 401, the first control unit 316 lights up when there is no abnormality in all the detection areas of the image display unit 401 that is the lighting target of the OK light 414 and the NG light 415.
[0089] The NG light 415 is a light area that is lit by the first control unit 316 when the result of the determination of the presence or absence of an abnormality in the designated detection area is abnormal in each image display unit 401 in the selected state. For example, when the NG light 415 is in the image display unit 401 in the selected state, when a detection area among the detection areas of the image display unit 401 that is the lighting target of the OK light 414 and the NG light 415 is determined to be abnormal, the detection area is lit by the first control unit 316.
[0090] The status display units 421a to 421f are display units that indicate the status of the camera device 2 and the virtual camera 5 corresponding to each image display unit 401. For example, the status display units 421a to 421f display contents indicating five statuses of the camera device 2 and the virtual camera, such as "not connected", "not set", "set up", and "monitoring". The so-called "not connected" means that the camera device 2 or the virtual camera 5 corresponding to the image display unit 401 is not connected to the information processing device 3, and no image data is displayed on the image display unit 401. For example, Figure 5 In the example shown, the camera 2 (specifically, the camera 2c) corresponding to the image display unit 401c is not connected to the information processing device 3, so the status display unit 421c indicates "not connected", and the image display unit 401c displays a state where no image data is displayed. The so-called "not set" means a state where the detection area, etc. for the image data displayed on the image display unit 401 has not been set. "Setting completed" refers to a state where the setting of the detection area, etc. for the image data displayed on the image display unit 401 is completed. "Monitoring" refers to a state where an image judgment action (monitoring action) is being performed based on the information of each detection field that has been set. When there is no need to distinguish or collectively refer to the status display units 421a to 421f, they are simply referred to as "status display unit 421".
[0091] [Configuration of the setting screen]
[0092] Figure 6 FIG. 1 is a diagram showing an example of a state in which a setting screen displayed on a display device of an information processing device according to an embodiment displays image data. Figure 6 The configuration of the setting screen 500 will be described.
[0093] The setting screen 500 is a screen for setting the designation of a detection area and the sensitivity and threshold value of image determination in the detection area. Figure 6 The setting screen 500 shown is a screen for setting the detection function, including a camera selection card 501, a setting image display unit 502, a play button 503a, a pause button 503b, a fast forward button 503c, a fast rewind button 503d, a stop button 503e, a drag bar 504, a slider 505, a time display unit 506, a setting read button 511, a setting write button 512, a setting reflection button 513, a close button 514, a detection area information unit 561, and a detection area button group 562.
[0094] The camera selection card 501 is an operation unit for selecting which of the image data of the virtual camera 5 and the image data of the camera devices 2b to 2f of the specified detection area desired by the user is to be displayed on the setting image display unit 502. For example, the image data output by the virtual camera 5 and the image data captured by the camera devices 2b to 2f correspond to "1" to "6" respectively. Figure 6 In the example, it indicates that "6", i.e., the image data of the camera device 2f, is selected.
[0095] The setting image display unit 502 is a display unit that displays image data acquired from the storage unit 302 based on an operation signal from the input unit 303 operated by the user via the first playback unit 304 . Figure 6 In the example of FIG. 5 , when the camera selection card 501 “6” is selected by the operation of the input unit 303 by the user, the first playback unit 304 is set to play and display the image data captured by the camera device 2f on the image display unit 502.
[0096] When the play button 503a is pressed by the user's operation of the input unit 303, the first playback unit 304 starts playing the image data displayed on the setting image display unit 502. When the pause button 503b is pressed by the user's operation of the input unit 303, the first playback unit 304 pauses the image data currently being played and displayed on the setting image display unit 502. When the fast forward button 503c is pressed by the user's operation of the input unit 303, the first playback unit 304 fast-forwards and displays the image data currently being played and displayed on the setting image display unit 502. When the fast rewind button 503d is pressed by the user's operation of the input unit 303, the first playback unit 304 fast-rewinds and displays the image data currently being played and displayed on the setting image display unit 502. When the stop button 503e is pressed by the user's operation of the input unit 303, the first playback unit 304 stops playing and displaying the image data currently being played and displayed on the setting image display unit 502.
[0097] The drag bar 504 is a bar-shaped body that displays the time corresponding to the image data being played and displayed on the setting image display unit 502 by the position of the slider 505 provided thereon. The slider 505 is a body and an operating unit that is slid to a position on the drag bar 504 corresponding to the shooting time of the image data being played and displayed on the setting image display unit 502 (the shooting time of the frame being displayed, or the acquisition time of the image being displayed). Conversely, when the slider 505 is slid by the user's operation of the input unit 303, the first playback unit 304 displays a frame or an image as image data at the shooting time corresponding to the position of the slider 505 on the drag bar 504 on the setting image display unit 502.
[0098] The time display unit 506 is a display unit that displays the shooting date (including the concept of the acquisition date) and the shooting time of the image data played back and displayed on the setting image display unit 502 by the first playback unit 304 .
[0099] When the setting read button 511 is pressed by the user's operation of the input unit 303, the setting unit 308 reads the image pickup device 2 or virtual camera 5 selected by the camera selection card 501 stored in the storage unit 302, and displays it in each display portion of the setting screen 500. When the setting write button 512 is pressed by the user's operation of the input unit 303, the setting unit 308 associates the information of the detection area set on the setting screen 500 with the image pickup device 2 or virtual camera 5 selected in the camera selection card 501, and stores it in the storage unit 302. When the setting reflect button 513 is pressed by the user's operation of the input unit 303, the setting unit 308 makes the information of the detection area set on the setting screen 500 effective (for example, temporarily stored in the RAM 103) until the application programs such as the observation screen 400 and the setting screen 500 are closed.
[0100] When the close button 514 is pressed by the user's operation of the input unit 303 , the setting unit 308 closes the setting screen 500 and displays the observation screen 400 on the display unit 333 .
[0101] The detection area information section 561 is a section that displays the name, sensitivity, threshold, and monitoring method (in Figure 6 "Monitoring") and the presence or absence of action (in Figure 6 The display area indicated as "action") (hereinafter, such information of the detection area is referred to as attribute information) and the operation area is selected.
[0102] The detection area button group 562 is a button group for editing the attribute information of the detection area specified by the detection area specifying unit 309 and displayed by the detection area information unit 561. The detection area button group 562 includes an action button 562a, a delete button 562c, a sensitivity increase button 562e, a sensitivity decrease button 562f, a threshold increase button 562g, and a threshold decrease button 562h. The functions of each button of the detection area button group 562 will be described below.
[0103] [Specified testing areas]
[0104] Figure 7 FIG. 1 is a schematic diagram showing an example of designating a detection area on a setting image display unit in a setting screen displayed on a display device of an information processing device according to an embodiment. Figure 7 This section describes the designation of the detection area.
[0105] Figure 7 The setting screen 500 shown shows a state where the detection area 570 is designated on the setting image display unit 502 that is playing and displaying the image data by the first playback unit 304. Specifically, first, when the user operates the input unit 303 and the pause button 503b is pressed at a time point when the image portion that requires image judgment of the detection function is displayed during the playback and display period of the image data on the setting image display unit 502, the first playback unit 304 pauses the image data that is being played and displayed on the setting image display unit 502, and displays the shooting date and shooting time of the frame or image at that time on the time display unit 506.
[0106] Next, in response to the user's operation of the input unit 303 (eg, a drag operation of the mouse 108 ), the detection region designation unit 309 designates and displays the detection area 570 on the setting image display unit 502 .
[0107] As the attribute information of the detection area 570, the setting unit 308 displays the name of the detection area 570 specified by the detection area specifying unit 309 on the detection area information unit 561, and the specified sensitivity (e.g., "50"), the specified threshold (e.g., "20%), the monitoring method (e.g., "grayscale"), and the presence or absence of action (e.g., "none") used as the default value in the image determination described later are displayed in the information area information unit 561. As the name of the detection area 570, the setting unit 308 displays, for example, Figure 7As shown in FIG. 1 , the detection area specified by the image data of the camera (camera 2f) "6" is named "A61". The detection area specifying unit 309 extracts the image within the detection area 570 in the frame or image displayed on the setting image display unit 502 at the time when the detection area 570 is specified as the detection reference image, and compares it with the camera 2 or virtual camera 5 ( Figure 7 In the example of the camera selection card 501, the camera 2f is associated with the camera 2f and stored in the storage unit 302, and the information of the designated detection area 570 (position information such as position and shape, and attribute information, etc.) is stored in the storage unit 302. Furthermore, the detection area designation unit 309 associates the information of the time when the detection area 570 is designated (time information) with the camera 2 or virtual camera 5 selected in the camera selection card 501 and stores it in the storage unit 302. In the same way as described above, multiple detection areas can be designated by the detection area designation unit 309.
[0108] exist Figure 7 The detection area ( Figure 7 First, through the operation of the user's input unit 303, one of the detection areas ( Figure 7 After that, the setting unit 308 makes the displayed part of the attribute information of the detection area of the selected detection area information unit 561 inverted. Figure 7 Detection area 570 is shown (often detection area name is "A61").
[0109] Then, when the user operates the input unit 303 and presses the action button 562a, the setting unit 308 switches the setting value of the action of the selected detection area 570. For example, when the action of the detection area 570 is "no", the setting unit 308 sets the action of the detection area 570 to "yes" after the operation button 562a is pressed. On the contrary, when the action of the detection area 570 is "no", the setting unit 308 sets the action of the detection area 570 to "no" after the operation button 562a is pressed. As described later, when the action is "yes", if the detection area determination unit 314 determines that the image of the detection area is abnormal, the external output unit 332 outputs an abnormality signal.
[0110] For example, when the user operates the input unit 303 and presses the delete button 562c, the setting unit 308 deletes the attribute information of the detection area 570 displayed on the detection area information unit 561, and deletes the information of the detection area 570 stored in the storage unit 302. In addition, the detection area designation unit 309 deletes the detection area 570 displayed on the setting image display unit 502, and deletes the detection reference image stored in the storage unit 302.
[0111] For example, when the user operates the input unit 303 and presses the sensitivity increase button 562e, the setting unit 308 increases the sensitivity value for image determination in the selected detection area 570 by a predetermined value. Conversely, when the user operates the input unit 303 and presses the sensitivity decrease button 562f, the setting unit 308 decreases the sensitivity value for image determination in the selected detection area 570 by a predetermined value.
[0112] For example, when the user operates the input unit 303 and presses the threshold value increase button 562g, the setting unit 308 increases the threshold value for image determination in the selected detection area 570 by a predetermined value. Conversely, when the user operates the input unit 303 and presses the threshold value decrease button 562h, the setting unit 308 decreases the threshold value for image determination in the selected detection area 570 by a predetermined value.
[0113] [Observe the image on the screen to judge the action]
[0114] Figure 8 This is a diagram showing an example of a state in which a monitoring operation is performed on an observation screen displayed on a display device of the information processing device according to the embodiment. Fig. 9 This is a schematic diagram showing an example of performing image determination in a detection area in a monitoring operation performed on an observation screen displayed on a display device of the information processing device according to the embodiment. Figure 8 and Fig. 9 This is a schematic diagram showing an example of performing image determination in a detection area in a monitoring operation performed on an observation screen displayed on the display device of the information processing device 3 according to the embodiment.
[0115] First reference Figure 8 , describing an overview of the image determination action (monitoring action) in the observation screen 400. Figure 8The observation screen 400 shown displays the images received from each camera device 2 and the virtual camera 5 in real time, and specifies the state of the detection area "A61" in the image display unit 401f. However, in the image judgment action of the observation screen 400, in order to avoid a cluttered appearance, the name of each judgment area is omitted from display on the image display unit 401. Here, the detection function refers to a function of performing image judgment of whether the image in the detection area is abnormal during the monitoring action. After transferring from the setting screen 500 to the observation screen 400, the setting of each detection area of the image display unit 401f is completed, and therefore, the first control unit 316 displays "setting completed" on the status display unit 421f.
[0116] Then, the user operates the input unit 303 to press the image display unit 401 corresponding to the photographic device 2 or virtual camera 5 for which the image judgment action (monitoring action) needs to be started, so that the image display unit 401 becomes a selected state. Figure 8 The example shows a state where the image display unit 401f is pressed and becomes a selected state. Then, when the monitoring start button 411 is pressed by the user operating the input unit 303, the first control unit 316 starts the image judgment operation of the real-time image data of the camera device 2f displayed on the image display unit 401f. At this time, the first control unit 316 Figure 8 As shown, "monitoring" is displayed in the status display unit 421f to indicate that the image judgment action is being performed on the camera 2f. The first control unit 316 also stores the information (moment information) of the start time of the image judgment action (monitoring action) in association with the camera 2 corresponding to the image display unit 401 in the selected state in the storage unit 302. When the stop monitoring button 412 is pressed by the user's operation of the input unit 303, the first control unit 316 stops the image judgment action of the image display unit 401 in the selected state, and stores the information (moment information) of the stop time of the image judgment action (monitoring action) in association with the camera 2 or virtual camera 5 corresponding to the image display unit 401 in the storage unit 302.
[0117] In the specific image determination operation on the observation screen 400, first, the image distribution unit 312 distributes the real-time image data received from the image receiving unit 301 to the image display units 401a to 401f for display. The detection area determination unit 314 obtains the designated detection area ( Figure 7 During the monitoring operation, the detection area determination unit 314 compares the detection reference image with the image of the detection area in the frame or image as image data in the designated detection area to perform image determination of abnormality.
[0118] Specifically, the detection area judgment unit 314 compares the image of the detection area in the frame or image corresponding to the detection area 570 with the detection reference image to judge whether the image is abnormal or not. The image judgment at this time is performed as follows, that is, the detection area judgment unit 314 compares the corresponding pixels between the detection reference image and the image of the detection area to judge whether the total number of pixels whose pixel values differ by more than the sensitivity of the attribute information of the detection area 570, that is, the difference, is greater than the threshold value of the attribute information (for example, when the threshold value is "20%", the number of pixels of 20% of the number of pixels in the image of the entire detection area (or the detection reference image)). If it is greater than the threshold value, the image of the detection area 570 is judged to be abnormal. In the comparison between the detection reference image and the image of the detection area, as long as the total number of pixels whose pixel values differ by more than the sensitivity of the attribute information of the detection area 570, that is, the difference, is less than the threshold value of the attribute information, then the image of the detection area 570 judged by the detection area judgment unit 314 is normal.
[0119] When the image judgment result of the detection area judgment unit 314 is a normal detection, if the image display unit 401f is the lighting target of the OK light 414 and the NG light 415, and the image judgment in the detection area of the image display unit 401 which is the lighting target of other lights is a normal detection, the first control unit 316 turns on the OK light 414.
[0120] In the case where the image determination result of the detection area determination unit 314 is abnormal detection, Fig. 9 As shown, the first control unit 316 displays the detection area 570 with a frame that is different from the normal color and thicker than the normal state. For example, in order to indicate abnormal detection, the first control unit 316 displays the frame representing the detection area 570 in "red" and thick lines, and keeps this display state for, for example, 1 second. When the image of the detection area 570 is determined as abnormal detection, the first control unit 316 turns on the NG light 415 when the image display unit 401f becomes the lighting target of the OK light 414 and the NG light 415. Then, after 1 second, the first control unit 316 returns the display of the frame representing the detection area 570 to the normal frame display. In addition, in the comparison between the detection reference image and the image of the detection area by the detection area determination unit 314, as long as the difference between the pixel values of the two is the sum of the number of pixels greater than the sensitivity of the attribute information of the detection area 570, that is, the difference value, is greater than the threshold value of the attribute information, the first control unit 316 continues to display the frame representing the detection area 570 in "red" and thick lines to continue to indicate the abnormal detection state. In this case, the lighting operation of the NG lamp 415 is also the same.
[0121] Then, the detection area determination unit 314 associates the information (time information) of the detection time when the abnormality is detected in the image determination operation with the camera 2 or virtual camera 5 that performs the image determination operation and stores it in the storage unit 302. The detection area determination unit 314 also associates the information (time information) of the detection time when the abnormality is detected in the image determination operation with the camera 2 or virtual camera 5 that performs the image determination operation and writes it in the tag file of the storage unit 302.
[0122] Then, when the detection area determination unit 314 determines that there is an abnormality, the external output unit 332 outputs an abnormality signal to the external device 10 .
[0123] As described above, when detecting an abnormality, the first control unit 316 displays the frame representing each detection area with a color and frame thickness different from the normal state. In this way, the user can visually grasp the time point in which the abnormality is detected in the detection area of each image display unit 401 during the image judgment operation (monitoring operation) of each image display unit 401.
[0124] [About the configuration when the virtual camera is an image judgment system]
[0125] Fig.10 This is an example in which a virtual camera is used as the image determination system in the overall configuration of the image determination system according to the embodiment. Fig.11 FIG. 1 is a schematic diagram showing an example of a state in which an observation screen output from an image determination system as a virtual camera is displayed on an observation screen displayed by a display device of an information processing device according to an embodiment. Fig.10 and Fig.11 , the structure and operation of the image determination system 1 according to the present embodiment when connected to another image determination system as the virtual camera 5 as described above will be described in detail.
[0126] like Fig.10 As shown, in the image determination system 1 according to the present embodiment, another image determination system 5a having a configuration equivalent to the image determination system 1 is provided as Figure 1The virtual camera 5 shown is connected to the information processing device 3 via the hub 4. The information processing device included in the image judgment system 5a (equivalent to the information processing device 3 of the image judgment system 1) is the same as the image judgment system 1, and receives image data from multiple IP cameras (equivalent to the photographic device 2 of the image judgment system 1), displays it on a screen equivalent to, for example, the observation screen 400 on a display (equivalent to the display 105) as a display device, and performs image judgment on the images received from each IP camera in the same manner as the image judgment system 1. In addition, an application (hereinafter referred to as the "sending application") is installed in the information processing device included in the image judgment system 5a, which is used to shoot the screen displayed on the display (e.g., the screen equivalent to the observation screen 400), convert it into an animation according to a video compression standard such as MotionJPEG or H.264, and send it to the outside. Here, as Fig.11 As shown, the information processing device included in the image determination system 5a uses the function of sending the application program to shoot the observation screen 900 (corresponding to the observation screen 400) displayed on the monitor and send the transformed animation (an example of the second image data) to the information processing device 3 via the hub 4. In other words, although the image determination system 5a itself is not a camera device, it can output the animation displayed on the monitor, so it can perform image determination by receiving the animation as image data equivalent to the camera device 2. The image determination system 5a is equivalent to the "another device" of the present invention.
[0127] For example Fig.11 As shown, in the observation screen 400 of the image determination system 1, the detection area designation unit 309 designates detection areas 571 and 572 as detection areas for abnormality detection for the animation (shot animation) of the observation screen 900 output from the image determination system 5a displayed in the image display unit 401a. Fig.11 For convenience of explanation, the OK lamp and the NG lamp included in the observation screen 900 are shown in a scaled (enlarged) manner on the image display unit 401a of the observation screen 400.
[0128] The detection area 571 is designated as including the OK light of the observation screen 900 (equivalent to the OK light 414 of the observation screen 400). The detection area 572 is designated as including the NG light of the observation screen 900 (equivalent to the NG light 415 of the observation screen 400). Then, the detection area designation unit 309 extracts the image in the detection area 571 when the OK light of the observation screen 900 is off as the detection reference image and saves it in the storage unit 302, and saves the information of the designated detection area 571 (position information such as position and shape, and attribute information, etc.) in the storage unit 302. The detection area designation unit 309 also extracts the image in the detection area 572 when the NG light of the observation screen 900 is off as the detection reference image and saves it in the storage unit 302, and saves the information of the designated detection area 572 (position information such as position and shape, and attribute information, etc.) in the storage unit 302.
[0129] Fig.11 The observation screen 400 shown in the figure shows that the image display unit 401a displays the animation received from the virtual camera 5, that is, the image determination system 5a in real time, and specifies the state of the detection areas 571 and 572. Next, the user presses the image display unit 401 (here, the image display unit 401a) corresponding to the photographing device 2 or the virtual camera 5 for which the image determination action (monitoring action) needs to be started through the operation of the input unit 303, and sets it to the selected state. Then, the first control unit 316 starts the image determination action for the real-time animation (shooting animation) of the image determination system 5a displayed on the image display unit 401a after the monitoring start button 411 is pressed through the operation of the user's input unit 303. The first control unit 316 also associates the information (time information) of the time when the image determination action (monitoring action) is started with the image determination system 5a corresponding to the image display unit 401a in the selected state and stores it in the storage unit 302. In addition, when the stop monitoring button 412 is pressed through the operation of the user's input unit 303, the first control unit 316 stops the image judgment action on the image display unit 401a in the selected state, and associates the information (time information) of the time when the image judgment action (monitoring action) is stopped with the image judgment system 5a corresponding to the image display unit 401a and saves it in the storage unit 302.
[0130] Then, the detection area determination unit 314 obtains the attribute information of the designated detection areas 571 and 572 from the storage unit 302. During the monitoring operation, the detection area determination unit 314 compares the detection reference image with the image of the detection area in the animation in the designated detection areas 571 and 572 to perform image determination.
[0131] Specifically, the detection area determination unit 314 compares the image of the detection area in the animation corresponding to the detection areas 571 and 572 with the detection reference image to perform image determination. At this time, the image determination is performed as follows. In the comparison between the detection reference image and the image of the detection area, when the difference between the two, that is, the sum of the number of pixels whose pixel values are greater than the sensitivity of the attribute information of the detection area 571, is greater than the threshold of the attribute information, the detection area determination unit 314 determines that the OK light is on as an image determination of the detection area 571. In the comparison between the detection reference image and the image of the detection area, as long as the difference between the two, that is, the sum of the number of pixels whose pixel values are greater than the sensitivity of the attribute information of the detection area 571, is less than the threshold of the attribute information, the detection area determination unit 314 determines that the OK light is not on as an image determination of the detection area 571.
[0132] Similarly, when the difference between the detection reference image and the image of the detection area, that is, the sum of the number of pixels whose pixel values differ by more than the sensitivity of the attribute information of the detection area 572, is greater than the threshold of the attribute information, the detection area determination unit 314 determines that the NG lamp is on as the image determination of the detection area 572. When the difference between the detection reference image and the image of the detection area, that is, the sum of the number of pixels whose pixel values differ by more than the sensitivity of the attribute information of the detection area 572, is less than the threshold of the attribute information, the detection area determination unit 314 determines that the NG lamp is not on as the image determination of the detection area 572.
[0133] Then, the detection area determination unit 314 associates the information (time information) of the detection time when the OK light or NG light is turned on during the image determination operation with the image determination system 5a, and stores it in the storage unit 302. The detection area determination unit 314 also associates the information (time information) of the detection time when the OK light or NG light is turned on during the image determination operation with the image determination system 5a, and writes it into the tag file of the storage unit 302.
[0134] Then, when the detection area determination unit 314 determines that the detection area determination unit 314 has made an abnormal determination (when the NG lamp lighting of the detection area 572 is detected), the external output unit 332 outputs an abnormality signal to the external device 10 .
[0135] As described above, when the image judgment system 5a serves as a virtual camera 5 that replaces the camera device 2 and is connected to the information processing device 3 via the hub 4, even if the usage and setting methods of the image judgment system 5a are not specifically understood, the image judgment of the animation output by the image judgment system 5a can be performed by the image judgment system 1, and the result of the image judgment recognized within the image judgment system 5a can be recognized by the image judgment system 1.
[0136] [About the multi-level structure when the virtual camera is an image judgment system]
[0137] Fig.12 Schematic diagram for explaining the state of an image determination system in which an image determination system as a virtual camera is implemented in a multi-stage connection. Fig.12 , the operation when other image determination systems are connected in multiple stages as virtual cameras 5 in the image determination system 1 involved in this embodiment is described in detail.
[0138] like Fig.10 and Fig.11 As described in the above, when the virtual camera 5 connected to the information processing device 3 is another image determination system having a configuration equivalent to the image determination system 1, the other image determination system can be further connected to other image determination systems as virtual cameras in multiple stages. Fig.12 In the embodiment, the image determination system 1 for displaying the observation screen 400 is connected to the image determination system 400_1 and the image determination system for displaying the observation screen 400_2 as virtual cameras. Furthermore, the image determination system for displaying the observation screen 400_1 is also connected to the image determination system for displaying the observation screen 400_1_1 as a virtual camera.
[0139] As described above, when the virtual camera connected to the image judgment system 1 is used as another image judgment system, the virtual camera corresponding to the other image judgment system can be further connected to the other image judgment system. Therefore, even if the use method and setting method of the multi-stage connected image judgment system are not understood, the OK light or NG light of the observation screen of the image judgment system can be used as the detection field, and the upper image judgment system can perform image judgment, thereby recognizing the image judgment result of the lowest image judgment system in the multi-stage structure through the image judgment of the captured animation by the upper image judgment system.
[0140] [Image judgment and playback operation on the browser screen]
[0141] Fig.13 This is a schematic diagram showing an example of a state in which recorded image data is played and displayed on a browser screen and displayed on a display device of the information processing device according to the embodiment. Fig.14 FIG. 1 is a schematic diagram showing an example of a state in which a mark list screen is displayed on a browser screen on a display device of an information processing device according to an embodiment. Fig.13 and Fig.14 The configuration of the browser screen 700 and the operation of displaying a plurality of recorded (saved) image data will be described.
[0142] The program (application) executed by the CPU 101, Fig.13 The browser screen 700 shown is displayed on the display 333. The browser screen 700 displays the image data received from the plurality of camera devices 2 by the second playback unit 318 and the image data output from the virtual camera 5, and reproduces the state of image judgment in each designated detection area through the past image data stored in the storage unit 302 by the image distribution unit 312.
[0143] The browser screen 700 includes image display portions 701 a to 701 f , a play button 703 a and a pause button 703 b , a fast forward button 703 c and a fast rewind button 703 d , a stop button 703 e , a drag bar 704 , a slider 705 , a time display portion 706 e , and a mark list button 707 .
[0144] The image display units 701a to 701f are display areas that display the past image data of each camera device 2 and the virtual camera 5 stored in the storage unit 302 through the second playback unit 318 according to the operation signal from the input unit 303 that is operated by the user. Here, it is assumed that the image display units 701a to 701f display image data of the same date and time. The image display units 701a to 701f can be selected when they are pressed by the user operating the input unit 303 (for example, a click operation of the mouse 108). When the image display units 701a to 701f are not distinguished or collectively referred to, they are simply referred to as "image display unit 701".
[0145] When the play button 703a is pressed by the user operation input unit 303, the second playback unit 318 starts to play the image data displayed on the image display unit 701. When the pause button 703b is pressed by the user operation input unit 303, the second playback unit 318 pauses the image data and actions displayed on the image display unit 701. When the fast forward button 703c is pressed by the user operation input unit 303, the second playback unit 318 fast forwards and displays the image data currently being played and displayed on the image display unit 701. When the fast rewind button 703d is pressed by the user operation input unit 303, the second playback unit 318 fast rewinds and displays the image data currently being played and displayed on the image display unit 701. When the stop button 703e is pressed by the user operation input unit 303, the second playback unit 318 stops playing and displaying the image data currently being played and displayed on the image display unit 701.
[0146] The drag bar 704 is a rod-shaped body that displays the time corresponding to the image data being played and displayed on the image display unit 701 by adjusting the position of the slider 705 provided thereon. The slider 705 is a body and an operation unit that is slid to a position on the drag bar 704 corresponding to the shooting time of the image data being played and displayed on the image display unit 701 (the shooting time of the frame being displayed, or the acquisition time of the image being displayed). Conversely, when the slider 705 is slid by the user's operation of the input unit 303, the second playback unit 318 displays a frame or an image as image data at the shooting time corresponding to the position of the slider 705 on the drag bar 704 on the image display unit 701.
[0147] The time display unit 706 is a display unit that displays the shooting date (including the concept of the acquired date) and the shooting time of the image data played back and displayed on the image display unit 701 by the second playback unit 318 .
[0148] When the mark list button 707 is pressed by the user operating the input unit 303, the list control unit 320 reads the mark file stored in the storage unit 302 and displays the contents of the mark file, thereby displaying the mark list. Fig.14 Tag list screen 800 is shown.
[0149] After the image display unit 701 of the browser screen 700 displays the past image data stored in the storage unit 302 via the second playback unit 318, the area display control unit 319 reads the information of the start and end time of the image determination action (monitoring action) stored in the storage unit 302, the information of the time when each detection area is designated, and the information of the time when each detection area is detected. Fig.13 As shown, the region display control unit 319 displays the frames representing the respective detection regions in an overlapping manner on the past image data displayed on the image display unit 701 based on the time information. For example, if the time when the respective detection regions are designated is before the time when the image determination operation starts based on the time information, the region display control unit 319 displays the frames representing the respective detection regions from the time when the image determination operation starts. The region display control unit 319 also deletes the displayed frames representing the respective detection regions when the time when the image data displayed on the image display unit 701 reaches the time when the image determination operation ends in the time information.
[0150] When the time when the image data displayed on the image display unit 701 reaches the abnormality detection time of the specific detection area in the time information, the area display control unit 319 displays a frame representing the detection area with a different color and thicker line frame than in the normal state. For example, the area display control unit 319 displays the frame representing each detection area in the same color as that displayed on the observation screen 400, for example, in "red" and thick line.
[0151] As described above, by controlling the display state of the frame of each detection area based on each time information stored in the storage unit 302, the area display control unit 319 can reproduce the state of the image determination operation (monitoring operation) in the browser screen 700 using the past image data. In this way, the user can check what image determination operation was performed in the past image data and images in addition to the current image data and images.
[0152] The above setting is that the image display units 701a~701f display image data of the same date and time, but it is not limited to this. For example, the image display units 701 each have a drag bar and an operation button, etc., and each image display unit 701 can display image data of a different date and time.
[0153] In addition, if Fig.14 As shown, when the mark list button 707 is pressed by the user's operation of the input unit 303, the list control unit 320 reads the mark file stored in the storage unit 302 and displays the mark list screen 800 showing the content of the mark file. Fig.14 As shown, the tag-sensitive screen 800 includes a detection history display section 802 (list).
[0154] The detection history display unit 802 is a list display area of the detection date and time based on the detection function. Specifically, the detection history display unit 802 displays the area name of the detection area, the detection date and time, and the name for identifying the corresponding camera 2 and virtual camera 5.
[0155] When a record of any detection field in the detection history display unit 802 is pressed by the user's operation of the input unit 303 , the list control unit 320 displays the image data of the date and time of the record display on the image display unit 701 in the second playback unit 318 .
[0156] Thus, by displaying Fig.14 The mark list screen 800 shown lists the date and time of the inspection in each inspection field, which can easily find out the time, trend or cause of the abnormality.
[0157] Furthermore, for the above Fig.12 As shown, in the case where the virtual camera 5 is multi-level connected to other image judgment systems, by pressing the record of the detection history display unit 802 of the observation screen 400, the second playback unit 318 can play and display the image data recorded on the observation screens 400_1, 400_2, and 400_1_1.
[0158] As described above, the image determination system 1 according to the present embodiment can be connected not only to a device that outputs directly captured image data such as the photographing device 2, but also to a device that outputs image data that can be used as a determination object such as the presence or absence of abnormalities, and performs image determination similar to the image data from the camera device on the image data obtained from the device. Thus, the image to be determined is not limited to the image data captured by the camera device, and images from devices other than the camera device can also be used as the determination object.
[0159] In addition, when the image judgment system 5a, which serves as a virtual camera 5 replacing the camera device 2, is connected to the information processing device 3 through the hub 4, even if the usage and setting methods of the image judgment system 5a are not specifically understood, the image judgment result recognized within the image judgment system 5a can be recognized by performing image judgment on the animation output by the image judgment system 5a on the output image judgment system 1, thereby enabling the image judgment system 1 to recognize the image judgment result.
[0160] Furthermore, when the virtual camera connected to the image judgment system 1 is used as another image judgment system, another image judgment system can be further connected as a virtual camera for the other image judgment system. Accordingly, even if the user does not know how to use and set up the image judgment systems connected in multiple stages, the user can use the OK light or NG light on the observation screen of the image judgment system as the detection field, and the upper image judgment system can perform image judgment on the photographic animation through the uppermost image judgment system 1, and recognize the image judgment result of the lowermost image judgment system in the multi-stage structure.
[0161] As mentioned above Fig.11 As shown, in the image recognition system 1 according to the present embodiment, the issuing application of the information processing device of the image recognition system 5a is set to shoot an animation of the entire screen displayed on the display (for example, the observation screen 900 displayed on the entire screen) and send it to the information processing device 3, but the present invention is not limited to this. For example, the issuing application of the information processing device of the image recognition system 5a may send a partial image of the screen displayed on the display (for example, a partial image including the OK light and the NG light of the observation screen 900), or an image formed by changing the layout of the partial images of the screen, etc. to the information processing device 3.
[0162] In addition, if Fig.10As shown, when the image determination system 5a is used as the virtual camera 5, a predetermined delay may occur when the information processing device of the image determination system 5a captures the screen of the display and sends the captured image to the information processing device 3 of the image determination system 1. In this case, the timing of the displayed image data and animation data will be different between the animation from the image determination system 5a and the image data from the camera 2 displayed on the observation screen 400. In this case, assuming that the above delay is known, the information processing device 3 of the image determination system 1 performs processing to stagger the display time of the image data and actions displayed on each image display unit 401 of the observation screen 400 in consideration of the delay, thereby unifying the timing of the images and animations displayed on each image display unit 401.
[0163] [Modification 1]
[0164] The following describes the image recognition system of the modified example 1 of the present embodiment, focusing on the differences from the image recognition system 1 involved in the present embodiment. In the present embodiment, the configuration and operation of connecting another image recognition system different from the image recognition system 1 as a virtual camera 5 to the information processing device 3 of the image recognition system 1 are described. In contrast, in the present modified example, the configuration and operation of connecting a system including a simulated camera as a virtual camera to the information processing device 3 will be described. The hardware configuration and functional module configuration of the information processing device 3 of the present modified example are the same as those described in the above embodiment.
[0165] Fig.15 FIG. 1 is a schematic diagram of the overall structure of an image recognition system according to Modification Example 1. Fig.15 , the structure and operation of the image determination system 1b involved in this modification example are described.
[0166] like Fig.15 As shown, the image determination system 1b is composed of a virtual camera 5b, which replaces Figure 1 The virtual camera 5b of the image determination system 1 shown in FIG. 5 is a system including an analog camera, specifically including an analog camera 21, a converter 22, and a PC (personal computer) 23 (an example of an information processing device). Fig.15 The illustration of the photographic device 2 is omitted. The virtual camera 5b is equivalent to "another device" of the present invention.
[0167] The analog camera 21 captures an image of a subject by converting light from the subject into an electrical signal, thereby generating an analog image signal.
[0168] The converter 22 is used to convert the analog image signal received by the analog camera 21 through the AV cable 24 into digital image data. The converter 22 sends the converted digital image data to the PC 23 through the USB (Universal Serial Bus) cable 25. The converter 22 is not limited to sending the image data to the PC 23 through the USB cable 25, and can also send it through, for example, an HDMI cable.
[0169] The PC 23 is an information processing device that performs predetermined analysis on the digital image data converted by the converter 22 to the analog signal captured by the analog camera 21. Fig.15 As shown, a parsing application 23a and a sending application 23b are installed.
[0170] The analysis application 23a is used to perform a predetermined analysis on the image data received from the converter 22. The analysis application 23a may not be used to analyze the image data, but may be used to simply play and display the image data.
[0171] The sending application 23b is used to capture the screen displayed on the monitor of the PC 23, convert it into an animation using an animation compression standard such as MotionJPEG or H.264, and transmit it to the information processing device 3 via the hub 4. That is, an animation (captured animation) (an example of second image data) of the display screen on the monitor showing the predetermined analysis result (e.g., including the playback display of image data) obtained by the analysis application 23a is transmitted from the network I / F of the PC 23 to the information processing device 3.
[0172] As described above, since an animation of a screen showing the analysis result of a predetermined analysis performed on the image data of the analog signal captured by the analog camera (for example, including the reproduction display of the image data) can be sent to the information processing device 3, it is possible to apply the following methods including: Fig.15 The simulated camera system is shown.
[0173] [Modification 2]
[0174] The following describes the image recognition system according to the second modification of the present embodiment, focusing on the differences from the image recognition system 1 according to the present embodiment. In the present embodiment, the configuration and operation of connecting another image recognition system different from the image recognition system 1 as a virtual camera 5 to the information processing device 3 of the image recognition system 1 are described. In contrast, in the present modification, the configuration and operation of connecting a system including an industrial digital camera as a virtual camera to the information processing device 3 are described. The hardware configuration and functional module configuration of the information processing device 3 according to the present modification are the same as those described in the above embodiment.
[0175] Fig.16 FIG. 2 is a schematic diagram showing the overall structure of an image recognition system according to Modification Example 2. Fig.16 , describing the structure and operation of the image determination system 1c involved in this variation.
[0176] like Fig.16 As shown, the image judgment system 1c has a virtual camera 5c which is used to replace Figure 1 The virtual camera 5 of the image determination system 1 shown in FIG. 5c is a system including industrial digital cameras, specifically including industrial digital cameras 31a, 31b (an example of a video camera), converters 32a, 32b, and PC 33 (an example of an information processing device). Fig.16 The illustration of the photographic device 2 is omitted. The virtual camera 5c is equivalent to "another device" of the present invention.
[0177] The industrial digital cameras 31a and 31b capture an image of the subject by converting light from the subject into an electrical signal, thereby generating image data of a moving image (eg, 10 [FPS]) composed of a plurality of frames (image data).
[0178] The converters 32a and 32b are used to convert the image data captured by the industrial digital cameras 31a and 31b and sent to the PC 33 via the HDMI cables 34a and 34b in accordance with the HDMI standard into image data that can be communicated in accordance with the USB standard. The converters 32a and 32b send the converted image data that can be communicated in accordance with the USB standard to the PC 33 via the USB cables 35a and 35b.
[0179] The PC 33 is an information processing device that performs predetermined analysis on the image data captured by the industrial digital cameras 31a and 31b. Fig.16 As shown, a parsing application 33a and a sending application 33b are installed.
[0180] The analysis application 33a is used to perform a predetermined analysis on the image data received from the converters 32a and 32b.
[0181] The sending application 33b is used to capture the screen displayed on the monitor of the PC 33, convert it into an animation using an animation compression standard such as MotionJPEG or H.264, and transmit it to the information processing device 3 through the hub 4. That is, an animation (captured animation) (an example of second image data) of the display screen on the monitor showing the predetermined analysis result (e.g., including the playback display of image data) obtained by the analysis application 33a is transmitted from the network I / F of the PC 33 to the information processing device 3.
[0182] As described above, since an animation of a screen showing the analysis result of a predetermined analysis performed on the image data of an analog signal captured by an industrial analog camera (for example, including a replay display of the image data) can be sent to the information processing device 3, it is possible to apply the following methods including: Fig.16 The simulated camera system is shown.
[0183] In addition, in the PC 33, when the function of the analysis application 33a is to analyze the image data received from the industrial digital cameras 31a and 31b without storing them in the storage device, etc., by connecting the system including the industrial digital cameras 31a and 31b as virtual cameras to the information processing device 3, the image data such as animation of the virtual camera can be stored in the storage unit 302. In addition, the state of the image judgment action (monitoring action) in the observation screen 400 can be reproduced by the area display control unit 319 using the image data such as animation stored in the storage unit 302. In this way, the user can confirm what kind of analysis action is performed by the analysis application 33a based on the image data previously taken by the virtual camera 5c.
[0184] Furthermore, even if one is not familiar with the system including the industrial digital cameras 31a and 31b, especially the usage and setting methods of the analysis application 33a, it is possible to identify the analysis results recognized in the virtual camera 5c by performing image judgment on the image data output by the virtual camera 5c in the information processing device 3.
[0185] [Variation 3]
[0186] The following describes an image recognition system according to a third variant of the present embodiment, focusing on the differences between the image recognition system 1 of the present embodiment. In the present embodiment, a configuration and operation of connecting another image recognition system different from the image recognition system 1 to the information processing device 3 of the image recognition system 1 as a virtual camera 5 is described. In contrast, in the present variant, a configuration and operation of connecting a system including an information processing device that displays the analysis results of an application program, etc., as a virtual camera to the information processing device 3 will be described. The hardware configuration and functional module configuration of the information processing device 3 according to the present variant are the same as those described in the above embodiment.
[0187] Fig.17 This is a schematic diagram of a software image that visualizes sound as an image output by a virtual camera connected to the image determination system according to Modification 3. Fig.17 , describing an example of using a system including an information processing device that displays analysis results of an application as a virtual camera.
[0188] Assume that the virtual camera involved in this variation includes a camera installed with Fig.17 An information processing device that performs wavelet transformation on waveform data such as vibration or sound shown in the figure and uses color to represent the waveform data. Then, a sending application installed in the information processing device transmits the waveform data such as vibration or sound to the image. Fig.17 The wavelet transformed waveform data shown in FIG. 1 is sent to the information processing device 3 via the hub 4 as an animation (an example of second image data) formed by photographing the application screen of the analysis application presented in color. As a result, the observation screen 400 displayed on the display unit 333 of the information processing device 3 can be displayed as follows: Fig.17 The application screen of the analysis application shown in FIG. 1 can be used to display the waveform on the application screen by specifying the detection area specifying unit 309. Fig.17 For example, in the setting contents of the detection area 573 (equivalent to the contents set on the setting screen 500), as shown in FIG. Fig.17 As shown in (a), since the red distribution is less in the waveform data of qualified products, Fig.17 As shown in (b), the waveform data of defective products has a large distribution of red, so the image with a small distribution of red is set as the detection reference image. Accordingly, in the image judgment operation of the observation screen 400, the waveform data of defective products can be detected by comparing the image of the detection area 573 with the above-mentioned detection reference image.
[0189] That is, even if one is not familiar with the method of using and setting the system including the information processing device that displays the analysis result of the application, the analysis result recognized by the analysis application can be recognized by performing image judgment on the image data output by the virtual camera in the information processing device 3. Fig.17 The analysis result of the application shown (ie, the analysis result within the virtual camera unit) can be cooperatively recognized through image judgment on the part of the information processing device 3 .
[0190] [Variation 4]
[0191] The following describes the image recognition system according to the modification example 4 of the present embodiment, focusing on the differences between the image recognition system 1 of the present embodiment. In the present embodiment, the configuration and operation of connecting another image recognition system different from the image recognition system 1 as a virtual camera 5 to the information processing device 3 of the image recognition system 1 are described. In the present modification example, the configuration and operation of connecting a smartphone that sends a video shot via wireless communication as a virtual camera to the information processing device 3 will be described. The hardware configuration and functional module configuration of the information processing device 3 according to the present modification example are the same as those described in the above embodiment.
[0192] Fig.18 FIG. 4 is a schematic diagram showing an overall structure of an image recognition system according to Modification 4. Fig.18 , describing the structure and operation of the image determination system 1d involved in this variation.
[0193] like Fig.18 As shown, the image determination system 1d is composed of a smartphone 41 (an example of a wireless communication terminal), and the smartphone 41 is used as a virtual camera 5d to replace Figure 1 The image determination system 1 shown in the figure has a virtual camera 5. The virtual camera 5d is equivalent to "another device" of the present invention.
[0194] The smartphone 41 is installed with a sending application for capturing the display screen, converting the display screen into an animation using an animation compression standard such as Motion JPEG or H.264, and transmitting it to the outside. The sending application transmits the captured animation (an example of the second image data) to the information processing device 3 via a wireless I / F possessed by the smartphone 41 using a communication standard such as Wi-Fi. The information processing device wirelessly connected to the information processing device 3 as the virtual camera 5d is not limited to the smartphone 41, and may also be, for example, a tablet terminal or a PC capable of wireless communication (an example of a wireless communication terminal).
[0195] As described above, since the screen displayed on the smartphone 41 can be sent to the information processing device 3, it is possible to Fig.17 The wireless communication terminal such as the smartphone 41 shown is used as a virtual camera.
[0196] For the screen of an application started and displayed by the smartphone 41, or the screen of a photo taken by the built-in camera of the smartphone 41, etc., the screen can be detected for abnormalities through image judgment on the part of the information processing device 3.
[0197] [Variation 5]
[0198] The following describes an image recognition system according to a third variant of the present embodiment, focusing on the differences between the image recognition system 1 of the present embodiment. In the present embodiment, a configuration and operation in which another image recognition system different from the image recognition system 1 is connected to the information processing device 3 of the image recognition system 1 as a virtual camera 5 is described. In contrast, in the present variant, a configuration and operation in which an information processing device (PC, etc.) that displays the results processed by software is connected to the information processing device 3 will be described. The hardware configuration and functional module configuration of the information processing device 3 according to the present variant are the same as those described in the above embodiment.
[0199] Fig.19This is a schematic diagram for explaining an image of a software image for creating a graph based on an image output by a virtual camera connected to the image determination system according to Modification 5. Fig.19 , describing an example in which an information processing device (PC, etc.) that displays results processed by software acts as a virtual camera.
[0200] The virtual camera involved in this modification is an information processing device running the following software, such as Fig.19 (a) and Fig.19 As shown in (b) of the software, you can input CSV (Comma-Separated Values) data generated by other applications, automatically create a chart, and update the updated CSV file data appropriately and reflect it in the chart. The software also has the following functions, such as changing the chart (such as Fig.19 Then, the sending application installed in the information processing device will display the Fig.19 The screen of the software for automatically creating and updating the chart shown in FIG. 1 is sent to the information processing device 3 via the hub 4 as a captured animation (an example of the second image data). As a result, the observation screen 400 displayed on the display unit 333 of the information processing device 3 is displayed. Fig.19 In this way, the area including the chart can be designated as the detection area by the detection area designation unit 309. In this case, the setting content of the detection area (equivalent to the content set on the setting screen 500) sets the image without red as the detection reference image. Thus, in the image judgment operation of the observation screen 400, it is possible to detect whether the value of the chart exceeds the threshold value by comparing the image of the detection area with the above-mentioned detection reference image.
[0201] As described above, since an animation of a display screen of the result displayed by an information processing device (PC, etc.) displaying the software processing result can be sent to the information processing device 3, the information processing device (PC, etc.) displaying the software processing result can be used as a virtual camera.
[0202] Furthermore, even if one is not familiar with the usage and setting methods of the information processing device (PC, etc.) that displays the software processing results, the result can be identified by judging the animated image output by the virtual camera of the result of the chart created by the software in the information processing device 3.
[0203] [Variation 6]
[0204] The following describes an image recognition system according to a third variant of the present embodiment, focusing on the differences between the image recognition system 1 of the present embodiment. In the present embodiment, a configuration and operation in which another image recognition system different from the image recognition system 1 is connected to the information processing device 3 of the image recognition system 1 as a virtual camera 5 will be described. In contrast, in the present variant, a configuration and operation in which a system including an information processing device that performs image processing on data of an image captured by a camera is connected to the information processing device 3 as a virtual camera will be described. The hardware configuration and functional module configuration of the information processing device 3 according to the present variant are the same as those described in the above embodiment.
[0205] Fig. 20 FIG. 1 is a schematic diagram of the overall structure of an image recognition system according to Modification Example 6. Fig. 20 The configuration and operation of the image determination system 1 e according to this modification will be described.
[0206] like Fig. 20 As shown in (a) and (b) of FIG. 5 , the image of the subject 61 captured by the camera 51 may become brighter or darker depending on the surrounding environment (weather, time, season, etc.). Fig. 20 In the example (a), when the camera 51 captures the subject 61 on a sunny day, a bright image is obtained. Fig. 20 As shown in (b), when the camera 51 captures the subject 61 in a rainy day, a dark image is obtained. In this case, when the image data captured by the camera 51 is directly sent to the information processing device 3, and a detection reference image set in the detection area specified on the setting screen 500 is fixed, an erroneous judgment result will be obtained when image judgment is performed on the image data captured in an environment different from the shooting environment of the image data.
[0207] To this end, the camera 51 is placed in a specific environment (such as Fig. 20 A certain part (partial image) of an image (first photographic image) of a subject 61 captured in a sunny environment (as shown in (a)) is registered as a reference part 62 (standard image) applicable to environmental adjustment. Then, after the reference part 62 is registered, the changes in brightness and saturation, etc. of the image corresponding to the reference part 62 are calculated from the image of the subject 61 captured by the camera device 51, and a transformation formula for converting the image into the brightness and saturation, etc. (i.e., color) of the reference part 62 is obtained. By applying the transformation formula to the entire image, an image with reduced influence of environmental changes can be obtained.
[0208] like Fig. 20 As shown in (c), the image determination system 1e in this modification example is composed of a virtual camera 5e, which is used to replace Figure 1The image determination system 1 shown in FIG. 1 has a virtual camera 5. The virtual camera 5e is a system that includes an information processing device that performs image processing on image data captured by a camera device, specifically including a camera device 51 and a PC 53 (an example of an information processing device). Fig. 20 The illustration of the camera device 2 is omitted in (c). In addition, the virtual camera 5e is equivalent to "another device" of the present invention.
[0209] The imaging device 51 is a digital camera that captures an image of a subject (photographing) by converting light reflected from the subject into an electrical signal, and generates image data of a moving image (eg, 10 [FPS]) composed of a plurality of frames (image data).
[0210] PC53 is an image processing device, which registers the reference portion 62 as described above, calculates the changes in brightness and saturation of the image corresponding to the reference portion 62 from the image (second camera image) taken by the camera device 51, finds a transformation formula, and performs image processing using the transformation formula on the entire image. Fig. 20 As shown, an image processing application 53a and a sending application 53b are installed.
[0211] The image processing application 53a is an application that executes the above-mentioned image processing.
[0212] The sending application 53b is used to capture the display screen, and convert the display screen into an animation using an animation compression specification such as Motion JPEG or H.264, and then transmit it to the information processing device 3. That is, an animation (captured animation) of the display screen of the display showing the image data that has been image processed by the image processing application 53a (an example of second image data) is sent from the network I / F of the PC 53 to the information processing device 3.
[0213] As described above, since the animation of the image screen that has been processed to reduce the effect of environmental changes on the image data taken by the camera device 51 can be sent to the information processing device 3, it is possible to use the information processing device 3 including the virtual camera. Fig. 20 The system includes an information processing device (PC53) that performs image processing on the image data captured by the camera 51 shown in (c). In this way, by performing image judgment on the image data with reduced influence of environmental changes on the information processing device 3 side, the occurrence of erroneous judgment can be suppressed.
[0214] The virtual camera connected to the information processing device 3 is not limited to the virtual cameras shown in the above-mentioned embodiment and each modified example. In addition, the following systems can also be used as virtual cameras. For example, a system for capturing temperature recording images for monitoring human detection and heat sources to prevent fires, a system for capturing depth sensor images for manual analysis, a system including a Raspberry Pi that acquires images via Wi-Fi, a system for visualizing radiation sensors for alarms, a system for detecting human skeleton information for operation analysis, a system for detecting human movement line information for operation analysis, a system for detecting human hand posture information for operating a PC, a system including a wearable camera for obtaining human line of sight, an alarm system for visualizing static electricity generation in a project using an electrostatic sensor to issue an alarm, a system for acquiring a PC screen image for business content detection and time analysis, and a system including a surveillance camera with no blind spots that can acquire multiple 360-degree images.
[0215] As in the above-described embodiment and each modified example, the display of real-time image data on the observation screen 400 and the display of past image data on the browser screen 700 are displayed separately, but the present invention is not limited thereto. In other words, the functions of the observation screen 400 and the browser screen 700 can be realized on the same screen.
[0216] In each display area (image display unit 401, 701), a frame is used as a display element representing each detection area, but the present invention is not limited to this, and any display element may be used as long as it represents each detection area.
[0217] The various functions of the above-mentioned embodiments and various modifications can be implemented by one or more processing circuits. Here, the so-called "processing circuit" refers to a processor that is programmed with various functions by software, such as a processor installed by electronic circuits, and an ASIC (Application Specific Integrated Circuit), DSP (Digital Signal Processor), FPGA (Field-Programmable Gate Array), SoC (System on a chip), GPU (Graphics Processing Unit) and traditional circuit modules and other devices designed to perform the above-mentioned functions.
[0218] The program running in the information processing device 3 in the above-mentioned embodiment and each variant example can be configured to be installed in a file in an installable format or an executable format on a computer-readable recording medium such as a CD-ROM (Compact Disc Read Only Memory), a floppy disk (FD), a CD-R (Compact Disk-Recordable), or a DVD (Digital Versatile Disk).
[0219] The program executed by the information processing device 3 of the above embodiment and each modified example can be stored in a computer connected to a network such as the Internet and downloaded from the network. The program executed by the information processing device 3 of the above embodiment and each modified example can also be provided or distributed via a network such as the Internet.
[0220] The program executed in the information processing device 3 of the above-mentioned embodiment and each modified example includes modules of the above-mentioned functional components. As actual hardware, the CPU (processor) reads and runs the program from the above-mentioned ROM, and calls the above-mentioned functional modules to the main storage device so that they are generated in each main storage device.
[0221] Explanation of symbols
[0222] 1, 1b to 1e image judgment system, 2, 2b to 2f camera device, 3 information processing device, 4 hub, 5 virtual camera, 5a image judgment system, 5b to 5e virtual camera, 10 external device, 21 analog camera, 22 converter, 23 PC, 23a analysis application, 23b transmission application, 24 AV cable, 25 USB cable, 31a, 31b industrial digital camera, 32a, 32b converter, 33 PC, 33a analysis application, 33b transmission application, 34a, 34b HDMI cable, 35a, 35b USB cable, 41 smart phone, 51 camera device, 53 PC, 53a image processing application, 53b transmission application, 61 object, 62 reference part, 101 CPU, 102 ROM, 103 RAM, 104 auxiliary memory, 105 display, 106 network I / F, 107 keyboard, 108 mouse, 109 DVD drive, 110 DVD, 111 peripheral I / F, 112 speaker, 113 bus, 301 image receiving unit, 302 storage unit, 303 input unit, 304 first playback unit, 308 setting unit, 309 detection area setting unit, 312 image allocation unit, 314 detection area determination unit, 316 first control unit, 318 second playback unit, 319 area display control unit, 320 list control unit, 321 second control unit, 331 display control unit, 332 external output unit, 333 display, 400, 400_1, 400_1_1, 400_2 observation screen, 401, 401a~401f image display unit, 411 monitoring start button, 412 monitoring stop button, 413 monitoring setting button, 414 OK light, 415 NG light, 421, 421a~421f status display unit, 500 setting screen, 501 camera selection card, 502 setting image display unit, 503a play button, 503b pause button, 503c fast forward button, 503d fast rewind button, 503e stop button, 504 drag bar, 505 slider, 506 time display unit, 511 setting read button, 512 setting write button, 513 setting reflection button, 514 close button, 561 detection area information unit, 562 detection area button group, 562a action button, 562c Delete button, 562e sensitivity button, 562f sensitivity button, 562g threshold increase button, 562h threshold decrease button, 570~573 detection area, 700 browser screen, 701, 701a~701f image display unit, 703a play button, 703b pause button, 703c fast forward button, 703d fast rewind button, 703e stop button, 704 drag bar, 705 slider, 706 time display unit, 707 mark list button, 800 mark list screen, 802 detection history display unit, 900 observation screen.
Claims
1. An image judging device for performing image judging on image data, comprising: a receiving unit configured to receive at least one of first image data and second image data, wherein the first image data is image data output from a camera device, and the second image data is image data output from another device, the other device being different from the camera device, the camera device being capable of outputting image data to be an image determination object, wherein the other device is an information processing device that performs a specified analysis process through application software, the receiving unit receives the second image data from the information processing device using an image including a result of the analysis process displayed on a display device of the information processing device as the second image data, wherein the analysis process includes a regenerated display of the image data; A first playing unit, configured to play and display at least one of the first image data and the second image data received by the receiving unit on a display device; a designating unit for designating a detection area of the image determination object in a setting screen display area of the display device that plays and displays the first image data or the second image data; and The determination unit is configured to perform the image determination on the first image data or the second image data within the detection area of the observation screen display area of the display device that displays at least one of the first image data and the second image data.
2. The image determination device according to claim 1, wherein: The judging unit generates a result of the image judging as a list, It further comprises a second playback unit for playing and displaying on the display device the first image data corresponding to the date and time of the result selected by operating the input device obtained from the list, or the second image data output by the information processing device.
3. The image determination device according to claim 1, wherein: The receiving unit receives a portion of image data on a screen displayed by a display device of the information processing device, or image data combined with the portion of image data.
4. The image determination device according to any one of claims 1 to 3, wherein: The system includes another image determination device that receives image data from one or more camera devices. The receiving unit receives the image data from the other image determination device as the second image data.
5. The image determination device according to claim 4, wherein: The system is a system in which other image determination devices including one or more image pickup devices are connected in multiple stages.
6. The image determination device according to any one of claims 1 to 3, wherein: The information processing device converts the image signal captured by the analog camera into digital image data through a converter, and outputs the image data including at least the digital image data as the second image data, The receiving unit receives the second image data from the information processing device.
7. The image determination device according to any one of claims 1 to 3, wherein: The information processing device performs a predetermined analysis process on the image data captured by the recorder, and the receiving unit receives the second image data from the information processing device using an image including a result of the analysis process displayed on a display device of the information processing device as the second image data.
8. The image determination device according to any one of claims 1 to 3, wherein: The information processing device is a wireless communication terminal that transmits at least a part of the image displayed on the display device to the image determination device via wireless communication. The receiving unit receives the at least a portion of the image from the wireless communication terminal as second image data through wireless communication.
9. The image determination device according to any one of claims 1 to 3, wherein: The information processing device updates and displays a graph based on data generated by the application software on the display device. The receiving unit receives the second image data from the information processing device, using the image including the chart displayed on the display device of the information processing device as the second image data.
10. The image determination device according to any one of claims 1 to 3, wherein: The information processing device uses a partial image of a first camera image previously captured by the camera device as a reference image, obtains a conversion formula for converting an image at a position corresponding to the partial image in a second camera image captured by the camera device into an image having the same color as the reference image, and outputs an image obtained by converting the second camera image using the conversion formula, wherein the conversion formula includes an animation compression specification. The receiving unit receives the image output from the information processing device and converted by the conversion formula as the second image data.
11. An image judgment system, comprising: at least one of a plurality of imaging devices that output first image data and a plurality of other devices that output second image data that is an image determination target and are different from the imaging devices, wherein: The other device is an information processing device that executes a specified parsing process by application software; and an image determination device for performing image determination on at least one of the first image data and the second image data, The image determination device comprises: a receiving unit configured to receive at least one of the first image data and the second image data, wherein the receiving unit receives the second image data from the information processing device using an image including a result of the analysis process displayed on a display device of the information processing device as the second image data, wherein the analysis process includes a regenerated display of the image data; A first playing unit, configured to play and display the at least one image data received by the receiving unit on a display device; a designating unit for designating a detection area of the image determination object in a setting screen display area of the display device that plays and displays the first image data or the second image data; and The determination unit is configured to perform the image determination on the first image data or the second image data within the detection area of the observation screen display area of the display device that displays at least one of the first image data and the second image data.
