Information processing system, information processing device, and information processing program

Abstract

To provide a status of a confirmation target such that, the status can be highly accurately confirmed.SOLUTION: An information processing system 1 includes a first information processing unit 10 connected to a photographing section 30 for acquiring a photographed image 40, and a second information processing unit 20 communicating with the first information processing unit 10. The second information processing unit 20 includes a reception section 28B, a conversion section 28c, and a display control section 28D. The reception section 28B receives the photographed image and digitized data obtained by converting a confirmation target image area, where a confirmation target C included in the photographed image is captured, according to a conversion rule. The conversion section 28C converts, according to the conversion rule, the digitized data into a status image representing a status of the confirmation target captured in the confirmation target image area. The display control section 28D displays a superimposed image obtained by superimposing the status image on the photographed image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] An embodiment of the present invention relates to an information processing system, an information processing device, and an information processing program. [Background technology]

[0002] There are known systems that remotely monitor a target object located in a remote location. For example, there are known systems that transmit captured images of the target object located in a remote location to a management server or the like and monitor the target object by checking the captured images on the management server side. There are also known systems that analyze the operational status of production equipment, which is an example of the target object, by analyzing the captured images. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2008-305259 A [Patent Document 2] JP 2002-132324 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in order to check the details of the situation of a remote location, it is necessary to analyze the captured images received at high image quality and high frame rate. However, depending on the communication conditions, degradation of image quality, low frame rate due to video delays and frame drops, etc., may occur, making it difficult to provide a highly accurate view of the situation of the target.

[0005] An object of the present invention is to provide an information processing system, an information processing device, and an information processing program that can provide a situation of a confirmation target in a manner that allows confirmation with high accuracy. [Means for solving the problem]

[0006] The information processing system of the embodiment is an information processing system including a first information processing device connected to an image capturing unit that captures a captured image, and a second information processing device that communicates with the first information processing device. The second information processing device includes a receiving unit, a converting unit, and a display control unit. The receiving unit receives the captured image and digitized data obtained by converting average values ​​of pixel values ​​constituting a confirmation target image area, which is included in the captured image and includes a confirmation target, according to a conversion rule. The converting unit converts the digitized data converted for each frame of the captured image into a situation image that represents the situation of the confirmation target that is captured in the confirmation target image area, based on the conversion rule. The display control unit displays a superimposed image in which the situation image is superimposed on the captured image. The conversion rule is represented by one threshold value for binarization and status information of the image area to be confirmed represented by each of the numerical values ​​after binarization of the average value of the pixel values ​​that make up the image area to be confirmed, the quantified data is binary data obtained by binarizing the average value of the pixel values ​​that make up the image area to be confirmed using the threshold value represented by the conversion rule, the object to be confirmed is a flashing light, and the status information represents whether the light is off or on. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a schematic diagram of an example of a configuration of an information processing system according to an embodiment. [Diagram 2] FIG. 2 is a schematic diagram of an example of a captured image. [Diagram 3] FIG. 3 is an explanatory diagram of an example of transmission control to the second information processing device by the communication control unit. [Figure 4A] FIG. 4A is a schematic diagram of an example of a superimposed image. [Figure 4B] FIG. 4B is a schematic diagram of an example of a superimposed image. [Diagram 5] FIG. 5 is a flowchart showing an example of the flow of information processing executed by the first information processing device. [Figure 6] FIG. 6 is a flowchart showing an example of the flow of information processing executed by the second information processing device. [Figure 7] FIG. 7 is a diagram showing the hardware configuration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, an information processing system, an information processing device, and an information processing program according to embodiments of the present invention will be described with reference to the drawings.

[0009] FIG. 1 is a schematic diagram of an example of a configuration of an information processing system 1 according to this embodiment.

[0010] The information processing system 1 includes a first information processing device 10, a second information processing device 20, and an image capturing unit 30. The first information processing device 10 and the second information processing device 20 are communicatively connected to each other via a network NW or the like. The first information processing device 10 and the image capturing unit 30 are communicatively connected directly to each other.

[0011] The first information processing device 10 is an information processing device disposed at a location that is a target of remote monitoring by the second information processing device 20. The first information processing device 10 is connected to an imaging unit 30. The first information processing device 10 may be configured to include the imaging unit 30.

[0012] The photographing unit 30 is a photographing device that obtains photographed image data by photographing. The photographing unit 30 is a known digital camera or the like. Hereinafter, the photographed image data may be simply referred to as a photographed image. The photographing unit 30 obtains a photographed image of the confirmation object C.

[0013] The confirmation target C is an object to be confirmed by a user who operates the second information processing device 20 that is located at a remote location relative to the first information processing device 10. The confirmation target C is an element in the real space that is the object of observation of at least two types of situation changes. In other words, the confirmation target C is an element in the real space where two or more types of situation changes occur.

[0014] Specifically, for example, the confirmation object C may be a light such as a flashing LED (Light-Emitting Diode), at least a partial area of ​​a display screen, a rotating part such as a discharger or turbine, a part such as a valve or dial that rotates or turns, water droplets such as water or oil, a specific part of a moving person or robot, a specified part of a remote-operated robot, etc.

[0015] In this embodiment, a case will be described in which the confirmation object C is a blinking LED provided on the facility M located at a location to be remotely monitored. The confirmation object C is set by an operation instruction by the user (described in detail later).

[0016] The photographing unit 30 photographs the confirmation object C continuously in chronological order, and sequentially outputs the photographed images obtained by photographing to the first information processing device 10 at the frame rate at which the images were photographed. Therefore, a photographed video composed of a plurality of photographed images is output from the photographing unit 30 to the first information processing device 10 at the frame rate at which the images were photographed. The frame rate at which the images are photographed by the photographing unit 30 is, for example, 30 fps to 60 fps, but is not limited to this range. It is preferable that the frame rate at which the images are photographed by the photographing unit 30 is 30 fps or more.

[0017] The photographing unit 30 may be fixed in real space or may be configured to be movable in real space as long as it can acquire a photographed image of the confirmation target C. In this embodiment, a form in which the photographing unit 30 is movably arranged by being supported by a user or a moving body such as a mobile robot will be described as an example.

[0018] The first information processing device 10 is an information processing device connected to the shooting unit 30, and is located in a remote location with respect to the second information processing device 20. The remote location means a location where the radio wave conditions during communication with the second information processing device 20 are not constant, and may include a communication period in which radio waves are weak and communication speed is low, or a communication period in which transmission delays, dropped frames of the captured images 40 to be transmitted, or the like occur, making real-time video communication difficult.

[0019] The first information processing device 10 includes a communication unit 12, a UI (user interface) unit 14, a storage unit 16, and a control unit 18. The communication unit 12, the UI unit 14, the storage unit 16, and the control unit 18 are communicatively connected via a bus or the like. The photographing unit 30 is communicatively connected to the control unit 18.

[0020] The communication unit 12 communicates with an external information processing device. In this embodiment, the communication unit 12 communicates with a second information processing device 20 via a network NW.

[0021] The UI unit 14 receives operation input from the user and outputs various information. The UI unit 14 includes an input unit 14A and a display unit 14B. The input unit 14A is an input device such as a keyboard that receives operation input from the user. The display unit 14B is a display that displays various information. The storage unit 16 stores various information. The storage unit 16 is, for example, a hard disk drive (HDD) or a memory.

[0022] The control unit 18 is a calculation unit that executes information processing.

[0023] The control unit 18 includes a setting unit 18A, an acquisition unit 18B, a generation unit 18C, and a communication control unit 18D. At least one of the setting unit 18A, the acquisition unit 18B, the generation unit 18C, and the communication control unit 18D is realized, for example, by one or more processors. For example, each of the above units may be realized by having a processor such as a CPU execute a program, that is, by software. Each of the above units may be realized by a processor such as a dedicated IC (Integrated Circuit), that is, by hardware. Each of the above units may be realized by using both software and hardware. When multiple processors are used, each processor may realize one of the units, or may realize two or more of the units.

[0024] The setting unit 18A sets a confirmation target image area in which the confirmation target C is captured in the captured image, and sets an irregular rule.

[0025] 2 is a schematic diagram of an example of a captured image 40 A. The captured image 40 A is an example of the captured image 40 captured by the imaging section 30.

[0026] The user operates the UI unit 14 while viewing the captured image 40A to select an area in which a desired confirmation target C appears as the confirmation target image area T. Fig. 2 shows a state in which areas of two blinking LEDs appearing in the captured image 40A have been selected as the confirmation target image area T1 and the confirmation target image area T2, respectively. The confirmation target image area T1 and the confirmation target image area T2 are examples of the confirmation target image area T.

[0027] By selecting a specific area included in the captured image 40A as the confirmation target image area T, the confirmation target image area T is set, and an object captured in the confirmation target image area T is set as the confirmation target C.

[0028] Continuing the explanation by returning to FIG 1, the setting unit 18A stores in the storage unit 16 a confirmation target image region T included in the photographed image 40.

[0029] Furthermore, the setting unit 18A sets a conversion rule for the confirmation target image region T included in the photographed image 40A.

[0030] The conversion rule is a conversion rule for converting the confirmation target image area T into a numerical value that represents the state of the confirmation target C reflected in the confirmation target image area T.

[0031] The situation of the confirmation target C reflected in the confirmation target image area T means the situation of the confirmation target C which is an element reflected in the confirmation target image area T and in which two or more types of situation changes occur.

[0032] Specifically, if the object to be checked C is a blinking LED, the status of the object to be checked C means, for example, "on" or "off." If the object to be checked C is a component such as a valve or dial that rotates or turns, the status of the object to be checked C means, for example, the degree of rotation or turning of the component. If the object to be checked C is a discharge, the status of the object to be checked C means, for example, the status of the discharge phenomenon. If the object to be checked C is droplets of water, oil, or the like, the status of the object to be checked C means, for example, the amount of droplets dripping or leaking.

[0033] The quantified data is data expressed by digital values ​​that represent the status of the confirmation target C captured in the confirmation target image area T. The digital value is, for example, a value represented by a color, a shape, or a judgment result such as a numerical value, but the numerical value is not limited to an integer. For example, the quantified data is P-value data obtained by converting the confirmation target image area T into a P value according to a conversion rule. P is an integer of 2 or more. For example, P corresponds to the number of types of changing status of the confirmation target C that represent two or more types of status changes.

[0034] When the quantified data is P-value data, the conversion rule is represented by P-1 threshold values ​​for P-value conversion and status information of the image area T to be confirmed represented by each of the numerical values ​​after P-value conversion of the image area T to be confirmed.

[0035] The setting unit 18A sets a conversion rule representing P-1 threshold values ​​for P-value conversion and the situation information of the image area T to be confirmed represented by each of the numerical values ​​after P-value conversion of the image area T to be confirmed, by accepting an instruction from a user to operate the input unit 14A, etc.

[0036] For example, assume that the status change of an LED, which is the confirmation target C reflected in the confirmation target image area T, represents two types of status changes, "on" and "off." Also assume that the binary data obtained by binarizing the confirmation target image area T to a numerical value "0" or a numerical value "1" representing the status of the confirmation target C reflected in the confirmation target image area T is used as the quantified data. In this case, for example, the setting unit 18A sets, as the conversion rule, a threshold value used for binarizing the average value of the pixel values ​​constituting the confirmation target image area T, and status information indicating that the binary value "0" represents the status of the confirmation target C "off" and the binary value "1" represents the status of the confirmation target C "on."

[0037] For example, the user operates the input unit 14A to input a conversion rule for converting the confirmation target image region T included in the captured image 40A into digitized data. In detail, the user inputs the conversion rule by inputting situation information of the confirmation target C represented by a threshold value used for binarization and a numerical value after binarization. The setting unit 18A sets the conversion rule by accepting the conversion rule input by the user's operation instruction of the input unit 14A.

[0038] The setting unit 18A stores the set conversion rule in the storage unit 16 in association with the set confirmation target image area T. The setting unit 18A also transmits the set conversion rule and the confirmation target image area T to the second information processing device 20. The second information processing device 20 receives the confirmation target image area T and the conversion rule, stores the received conversion rule and confirmation target image area T in association with each other, and uses them for various processes described later.

[0039] The setting of the confirmation target image area T and the conversion rule may be executed on the side of the second information processing device 20 described later. In this embodiment, a form in which the setting of the confirmation target image area T and the conversion rule is executed on the side of the second information processing device 20 described later will be described as an example.

[0040] In this case, the setting unit 18A of the first information processing device 10 transmits a captured image 40 captured by the imaging unit 30 to the first information processing device 10. Then, the setting unit 18A receives the confirmation target image area T and the conversion rule set using the captured image 40 from the second information processing device 20. The setting unit 18A sets the confirmation target image area T and the conversion rule by storing the received confirmation target image area T and the conversion rule in the storage unit 16 in association with each other.

[0041] The quantified data may be data expressed by a numerical value representing the status of the confirmation target C reflected in the confirmation target image area T, and is not limited to P-value data obtained by converting the confirmation target image area T into a P value according to a conversion rule. For example, the quantified data may be data obtained by converting the status of the confirmation target C reflected in the confirmation target image area T into a numerical value for each status according to a predetermined rule. In this case, the conversion rule may be a conversion rule for converting the status of the confirmation target image area T into the numerical value. In detail, in this case, the conversion rule may be represented by a rule for converting the status of the confirmation target image area T into a numerical value, and status information of the confirmation target C represented by each of the converted numerical values.

[0042] In this embodiment, a case will be described as an example where the status change of the LED, which is the confirmation target C, reflected in the confirmation target image area T, represents two types of status changes, "on" and "off." Also, a case will be described as an example where the binarized data, which is binarized to the numerical value "0" representing the status "off" of the confirmation target C reflected in the confirmation target image area T or the numerical value "1" representing the status "on," is used as the quantified data.

[0043] When the conversion rules are set in both the first information processing device 10 and the second information processing device 20, the acquisition unit 18B, the generation unit 18C, and the communication control unit 18D execute the following processes.

[0044] The acquiring unit 18B acquires the captured images 40 from the photographing unit 30. As described above, the photographing unit 30 sequentially outputs the captured images 40 obtained by photographing to the first information processing device 10 at the frame rate at the time of photographing. Therefore, the acquiring unit 18B sequentially acquires the multiple captured images 40 at the frame rate at the time of photographing by the photographing unit 30.

[0045] The generating unit 18C generates digitalized data by converting a confirmation target image region T, which is included in the photographed image 40 and in which the confirmation target C appears, according to a conversion rule.

[0046] The generating unit 18C reads the confirmation target image region T stored in the storage unit 16. Then, the generating unit 18C specifies the confirmation target image region T included in the acquired photographed image 40 by a known image processing technique using pattern matching, feature amount, etc. Then, the generating unit 18C converts the specified confirmation target image region T included in the photographed image 40 into digitized data in accordance with the conversion rule stored in the storage unit 16.

[0047] 2, a scene is assumed in which a confirmation target image region T1 and a confirmation target image region T2 included in a photographed image 40 are set as the confirmation target image region T. In this case, the generation unit 18C identifies each of the confirmation target image region T1 and the confirmation target image region T2 included in the photographed image 40 newly acquired by the acquisition unit 18B. Then, the generation unit 18C converts each of the identified confirmation target image region T1 and the confirmation target image region T2 into digitalized data in accordance with the conversion rule.

[0048] For example, assume that the conversion rule includes a threshold value used to binarize the average pixel values ​​that make up the image area T to be confirmed, and status information indicating that the binarized number "0" represents the status of the object C to be confirmed as "off" and that the binarized number "1" represents the status of the object C to be confirmed as "on."

[0049] In this case, the generation unit 18C binarizes the average pixel value of the identified confirmation target image area T included in the captured image 40 using a threshold value included in the conversion rule. Through this binarization conversion process, the generation unit 18C converts each of the confirmation target image area T1 and the confirmation target image area T2 included in the captured image 40 into a numerical value "0" or "1" representing the blinking state of the confirmation target C captured in each of the confirmation target image area T1 and the confirmation target image area T2. Through this conversion process, the generation unit 18C generates numerical data for each of the confirmation target image area T1 and the confirmation target image area T2.

[0050] Returning to Figure 1, we will continue with the explanation.

[0051] The communication control unit 18D transmits to the second information processing device 20 the captured image 40 acquired by the acquisition unit 18B and the digitized data generated by the generation unit 18C.

[0052] The communication control unit 18D sequentially transmits the captured images 40 to the second information processing device 20 at a frame rate lower than that at which the captured images 40 were captured, and sequentially transmits digitized data sequentially generated from each of the multiple captured images 40 to the second information processing device 20 at a frame rate higher than that of the captured images 40. That is, the communication control unit 18D sequentially transmits the captured images 40 to the second information processing device 20 at a frame rate lower than that at which the captured images 40 were captured, and sequentially transmits the digitized data to the second information processing device 20 at a frame rate higher than that of the captured images 40. In this embodiment, a form in which the communication control unit 18D sequentially transmits digitized data to the second information processing device 20 at the frame rate at which the captured images 40 were captured will be described as an example.

[0053] In detail, when the communication control unit 18D starts acquiring the captured images 40 sequentially transmitted from the photographing unit 30, it transmits one Nth frame of the captured image 40 to the second information processing device 20. N is an integer equal to or greater than 1. A frame means a frame, that is, one captured image 40. The Nth frame means the Nth captured image 40 (frame, frame) when counting the captured images 40 in chronological order from the first captured image 40 captured when photographing by the photographing unit 30 begins.

[0054] In addition, the communication control unit 18D sequentially transmits the quantified data generated by the generation unit 18C from the confirmation target image area T contained in each of the captured images 40 from the Nth frame to M frames (M is an integer greater than or equal to 2) to the second information processing device 20 at the frame rate when the captured images 40 were captured by the shooting unit 30.

[0055] Then, after transmitting the digitized data generated from the (N+M)th frame of the captured image 40 to the second information processing device 20, the communication control unit 18D sets N+M+1 to the new N and repeats the transmission of the new Nth frame of the captured image 40 to the second information processing device 20 and the transmission of the digitized data for M frames to the second information processing device 20.

[0056] Through these processes, the communication control unit 18D sequentially transmits to the second information processing device 20 digitized data of the confirmation target image region T included in each of the M frames of the captured images 40 sequentially acquired from the capturing unit 30, every time the communication control unit 18D transmits one frame of the captured image 40 acquired from the capturing unit 30 to the second information processing device 20. For this reason, the communication control unit 18D transmits the captured images 40 to the second information processing device 20 at a low frame rate, and transmits the digitized data to the second information processing device 20 at a high frame rate. The value of M may be determined in advance so that the captured images 40 are transmitted to the second information processing device 20 at a low frame rate, and the digitized data are transmitted to the second information processing device 20 at a high frame rate.

[0057] Fig. 3 is an explanatory diagram of an example of transmission control to the second information processing device 20 by the communication control unit 18D. In Fig. 3, digitized data 42A is an example of digitized data 42 for M frames of a confirmation target image area T1 included in the photographed image 40B. The digitized data 42A includes digitized data 42A1 to digitized data 42AM for M frames. The digitized data 42B is an example of digitized data 42 for M frames of a confirmation target image area T2 included in the photographed image 40B. The digitized data 42B includes digitized data 42B1 to digitized data 42BM for M frames.

[0058] Each time the communication control unit 18D transmits one frame of captured image 40B acquired from the imaging unit 30 to the second information processing device 20, it sequentially transmits M frames of digitized data 42A1 through 42AM and M frames of digitized data 42B1 through 42BM generated from each of the M frames of captured images 40 sequentially acquired from the imaging unit 30 to the second information processing device 20. For this reason, the communication control unit 18D transmits the captured image 40B to the second information processing device 20 at a low frame rate, and transmits the digitized data 42 to the second information processing device 20 at a high frame rate.

[0059] The communication control unit 18D may transmit the captured image 40 acquired from the imaging unit 30 to the second information processing device 20 after reducing the resolution of the image 40 to a value lower than the resolution at the time of imaging.

[0060] Continuing the explanation by returning to Fig. 1, the second information processing device 20 will now be explained.

[0061] The second information processing device 20 is an information processing device that is disposed in a remote location with respect to the first information processing device 10 and the confirmation target C. The second information processing device 20 is operated by a user who monitors or checks the confirmation target C, for example.

[0062] The second information processing device 20 includes a communication unit 22, a UI unit 24, a storage unit 26, and a control unit 28. The communication unit 22, the UI unit 24, the storage unit 26, and the control unit 28 are communicatively connected via a bus or the like.

[0063] The communication unit 22 communicates with an external information processing device. In this embodiment, the communication unit 22 communicates with the first information processing device 10 via the network NW.

[0064] The UI unit 24 receives operation input by the user and outputs various information. The UI unit 24 includes an input unit 24A and a display unit 24B. The input unit 24A is an input device such as a keyboard that receives operation input by the user. The display unit 24B is a display that displays various information. The storage unit 26 stores various information. The storage unit 26 is, for example, a HDD or a memory.

[0065] The control unit 28 is a calculation unit that executes information processing.

[0066] The control unit 28 includes a setting unit 28A, a receiving unit 28B, a conversion unit 28C, and a display control unit 28D. At least one of the setting unit 28A, the receiving unit 28B, the conversion unit 28C, and the display control unit 28D is realized, for example, by one or more processors. For example, each of the above units may be realized by having a processor such as a CPU execute a program, that is, by software. Each of the above units may be realized by a processor such as a dedicated IC (Integrated Circuit), that is, by hardware. Each of the above units may be realized by using both software and hardware. When multiple processors are used, each processor may realize one of the units, or may realize two or more of the units.

[0067] The setting unit 28A sets a confirmation target image region T in which the confirmation target C is captured in the photographed image 40, and sets an irregular rule.

[0068] The setting unit 28A acquires the captured image 40 from the first information processing device 10. The setting unit 28A displays the captured image 40 acquired from the first information processing device 10 on the display unit 24B.

[0069] The user operates the input unit 24A while viewing the displayed captured image 40 to select an area in which a desired confirmation target C appears as the confirmation target image area T. By selecting a predetermined area included in the captured image 40 as the confirmation target image area T, the confirmation target image area T is set, and the object appearing in the confirmation target image area T is set as the confirmation target C. The setting unit 28A stores the confirmation target image area T included in the captured image 40 in the memory unit 26.

[0070] Furthermore, the setting unit 28A sets a conversion rule for the confirmation target image region T included in the captured image 40. The setting unit 28A sets the conversion rule, which represents, for example, P-1 threshold values ​​for P-value conversion and situation information for the confirmation target image region T represented by each of the numerical values ​​after P-value conversion for the confirmation target image region T, by receiving an operation instruction from the user via the input unit 24A, etc.

[0071] As described above, in this embodiment, the setting unit 28A sets, as the conversion rule, a threshold value used for binarizing the average value of the pixel values ​​constituting the confirmation target image area T, and status information indicating that the binarized value "0" represents the status of the confirmation target C as "off" and that the binarized value "1" represents the status of the confirmation target C as "on."

[0072] For example, the user operates the input unit 24A to input a conversion rule for converting an area selected as the confirmation target image area T included in the captured image 40 into digitized data. In detail, the user inputs the conversion rule by inputting situation information of the confirmation target C represented by a threshold value used for binarization and a numerical value after binarization. The setting unit 28A sets the conversion rule by accepting the conversion rule input by the user through an instruction to operate the input unit 24A.

[0073] The setting unit 28A stores the set conversion rule in the storage unit 26 in association with the set confirmation target image area T. The setting unit 28A also transmits the set conversion rule and the confirmation target image area T to the first information processing device 10. The first information processing device 10 receives the confirmation target image area T and the conversion rule, and stores them in the storage unit 16 in association with each other.

[0074] As described above, the setting of the confirmation target image area T and the conversion rule may be performed by the setting unit 18A of the first information processing device 10. In this case, the setting unit 28A receives the confirmation target image area T and the conversion rule set by the setting unit 18A from the first information processing device 10. The setting unit 28A sets the confirmation target image area T and the conversion rule by storing the received confirmation target image area T and the conversion rule in the storage unit 26 in association with each other.

[0075] The receiving section 28B receives the captured image 40 and the digitized data 42 from the first information processing device 10.

[0076] In detail, the receiving unit 28B receives the captured image 40 at a frame rate lower than that at which the captured image 40 was captured by the imaging unit 30, and receives the digitized data 42 at a frame rate higher than that at which the captured image 40 was captured by the imaging unit 30. For example, the receiving unit 28B receives the digitized data 42 at the frame rate at which the captured image 40 was captured by the imaging unit 30.

[0077] Specifically, for example, each time the receiving unit 28B receives one frame of the captured image 40 from the first information processing device 10, it receives from the first information processing device 10 digitized data 42 that has been converted from the confirmation target image area T contained in each of the M frames of the captured images 40.

[0078] The conversion unit 28C converts the received digitized data 42 into a situation image based on the conversion rule every time it receives one frame of digitized data 42. The situation image is an image that represents the situation of the confirmation target C that is captured in the confirmation target image area T.

[0079] For example, assume that the conversion rule includes a threshold value used to binarize the average value of the pixel values ​​that make up the target image area T, and status information indicating that the binarized number "0" represents the status of the target C reflected in the target image area T as "off," and that the binarized number "1" represents the status of the target C reflected in the target image area T as "on."

[0080] In this case, when the numerical value represented by the digitized data 42 is "1," the conversion unit 28C determines that the status of the confirmation target C reflected in the confirmation target image area T represented by the digitized data 42 is "lit." Then, the conversion unit 28C generates a status image representing the status of the confirmation target C reflected in the confirmation target image area T as "lit."

[0081] Furthermore, when the numerical value represented by the digitized data 42 is "0," the conversion unit 28C determines that the status of the confirmation target C reflected in the confirmation target image area T represented by the digitized data 42 is "lights off." Then, the conversion unit 28C generates a status image representing the status of the confirmation target C reflected in the confirmation target image area T as "lights off."

[0082] The situation image may be an image that allows confirmation of the situation of the confirmation target C reflected in the confirmation target image area T. For example, a situation image that represents the situation "lit" is represented by at least one combination of highlighted images such as characters, colors, frames, etc., that represent the situation of the confirmation target C reflected in the confirmation target image area T being "lit". For example, a situation image that represents the situation "lights off" is represented by at least one combination of images such as characters, colors, frames, etc., that represent the situation of the confirmation target C reflected in the confirmation target image area T being "lights off".

[0083] Furthermore, the conversion unit 28C may generate a situation image that further includes identification information of each of the confirmation targets C captured in the confirmation target image area T.

[0084] The display control unit 28D generates a superimposed image by superimposing the situation image on the photographed image 40, and displays it on the display unit 24B.

[0085] 4A and 4B are schematic diagrams of an example of the superimposed image 46. FIG.

[0086] Fig. 4A is a schematic diagram of an example of a superimposed image 46A. The superimposed image 46A is an example of the superimposed image 46. Fig. 4A shows an example of the superimposed image 46A when the receiving unit 28B receives the photographed image 40B shown in Fig. 3, the digitized data 42A1, and the digitized data 42B1, and the converting unit 28C generates the situation image 44 from each of the digitized data 42A1 and the digitized data 42B1.

[0087] For example, the conversion unit 28C generates a situation image 44A representing the status "lit" of the confirmation target C reflected in the confirmation target image area T1 from the numerical value "1" represented by the digitized data 42A1. The conversion unit 28C also generates a situation image 44B representing the status "lit" of the confirmation target C reflected in the confirmation target image area T2 from the numerical value "1" represented by the digitized data 42B1. The situation image 44A and the situation image 44B are examples of the situation image 44. The conversion unit 28C may generate a situation image 44 including identification information of the confirmation target C reflected in each of the confirmation target image areas T1 and T2, as shown in FIG. 4A.

[0088] The display control unit 28D generates a superimposed image 46A by superimposing the situation image 44A and the situation image 44B on or near the confirmation target image region T1 and the confirmation target image region T2 included in the captured image 40 received simultaneously with or immediately before the digitized data 42A1 and the digitized data 42B1. Then, the display control unit 28D displays the superimposed image 46A on the display unit 24B.

[0089] Fig. 4B is a schematic diagram of an example of a superimposed image 46B. The superimposed image 46B is an example of the superimposed image 46. Fig. 4B shows an example of the superimposed image 46B when the receiving unit 28B receives the photographed image 40B shown in Fig. 3, the digitized data 42A2, and the digitized data 42B2, and the converting unit 28C generates the situation image 44 from each of the digitized data 42A2 and the digitized data 42B2.

[0090] For example, the conversion unit 28C generates a situation image 44A representing the situation "lights off" of the confirmation target C reflected in the confirmation target image area T1 from the numerical value "0" represented by the digitized data 42A2. The conversion unit 28C also generates a situation image 44B representing the situation "lights off" of the confirmation target C reflected in the confirmation target image area T2 from the numerical value "0" represented by the digitized data 42B2. The situation image 44A and the situation image 44B are examples of the situation image 44. The conversion unit 28C may generate a situation image 44 including identification information of the confirmation target C reflected in each of the confirmation target image areas T1 and T2, as shown in FIG. 4B.

[0091] The display control unit 28D generates a superimposed image 46B by superimposing the situation image 44A and the situation image 44B on or near the confirmation target image region T1 and the confirmation target image region T2 included in the captured image 40 received simultaneously with or immediately before the digitized data 42A2 and the digitized data 42B2. Then, the display control unit 28D displays the superimposed image 46B on the display unit 24B.

[0092] As a result, the display control unit 28D sequentially displays on the display unit 24B superimposed images 46 each of which is a situation image 44 generated based on a conversion rule from each of the quantified data 42 received at a high frame rate, on top of a captured image 40 received at a low frame rate.

[0093] That is, on the display unit 24B, a situation image 44 which changes sequentially according to the situation of the confirmation object C captured in the confirmation object image area T, represented by the digitized data 42 received at the high frame rate, is superimposed on the captured image 40 received at the low frame rate. For example, if the confirmation object C is a blinking LED, the display unit 24B displays an image of the situation image 44 which indicates that the blinking situation changes at a speed according to the high frame rate, on the captured image 40 received at the low frame rate.

[0094] Therefore, the user can accurately check the status of the confirmation object C located in a remote location by checking the superimposed image 46 displayed on the display unit 24B.

[0095] Next, an example of the flow of information processing executed by the information processing system 1 of this embodiment will be described.

[0096] It is assumed that the setting unit 18A of the first information processing device 10 and the setting unit 28A of the second information processing device 20 store the image area T to be confirmed and the conversion rule in correspondence with each other in the memory unit 16 of the first information processing device 10 and the memory unit 26 of the second information processing device 20, respectively.

[0097] FIG. 5 is a flowchart showing an example of the flow of information processing executed by the first information processing device 10. As shown in FIG.

[0098] The acquisition unit 18B of the first information processing device 10 starts acquiring the captured images 40 sequentially transmitted from the imaging unit 30 (step S100).

[0099] The communication control unit 18D transmits the Nth frame of the photographed image 40 received from the photographing unit 30 to the second information processing device 20 (step S102).

[0100] The generation unit 18C identifies the confirmation target image area T included in the Nth frame of the photographed image 40 received from the photographing unit 30 (step S104). The generation unit 18C reads the confirmation target image area T stored in the storage unit 16, and identifies an area included in the photographed image 40 that corresponds to the read confirmation target image area T, thereby identifying the confirmation target image area T included in the photographed image 40.

[0101] The generating unit 18C generates digitized data 42 by converting the confirmation target image region T identified in step S104 in accordance with the conversion rule stored in the storage unit 16 (step S106). When the conversion rule includes a threshold value for binarization, the generating unit 18C uses the threshold value to binarize the average value of each pixel value of one or more pixels constituting the identified confirmation target image region T, thereby generating the digitized data 42.

[0102] The communication control unit 18D transmits the digitized data 42 generated in step S106 to the second information processing device 20 (step S108).

[0103] The communication control unit 18D judges whether or not the digitized data 42 for M frames has been transmitted to the second information processing device 20 after transmitting the photographed image 40 to the second information processing device 20 (step S110). If a negative judgment is made in step S110 (step S110: No), N is counted up by 1 (step S112) and the process returns to step S104. As described above, N is an integer equal to or greater than 1, and M is an integer equal to or greater than 2.

[0104] If the determination in step S110 is affirmative (step S110: Yes), the process proceeds to step S114. In step S114, the control unit 18 determines whether or not to end the transmission process of the captured image 40 and the digitized data 42 (step S114). For example, the control unit 18 makes the determination in step S114 by determining whether or not an end signal has been received by a user's operation instruction on the input unit 14A or the like. Alternatively, for example, the control unit 18 may make the determination in step S114 by determining whether or not an end signal has been received from the second information processing device 20.

[0105] If the determination in step S114 is negative (step S114: No), the process proceeds to step S116. In step S116, N is updated to N+M+1 (step S116), and the process returns to step S102. If the determination in step S114 is positive (step S114: Yes), this routine ends.

[0106] Next, an example of the flow of information processing executed by the second information processing device 20 of this embodiment will be described.

[0107] FIG. 6 is a flowchart showing an example of the flow of information processing executed by the second information processing device 20 of this embodiment.

[0108] The receiving section 28B of the second information processing device 20 receives the photographed image 40 and the digitized data 42 from the first information processing device 10 (steps S200 and S202).

[0109] The conversion unit 28C converts the digitized data 42 received in step S202 into a situation image 44 based on the conversion rules (step S204).

[0110] The display control unit 28D generates a superimposed image 46 by superimposing the situation image 44 converted in step S204 on the captured image 40 received immediately before (step S206), and displays the generated superimposed image 46 on the display unit 24B (step S208).

[0111] Next, the receiving unit 28B judges whether or not the digitized data 42 has been received (step S210). If the determination in step S210 is affirmative (step S210: Yes), the process proceeds to step S204. If the determination in step S210 is negative (step S210: No), the receiving unit 28B judges whether or not the captured image 40 has been received (step S212). If the determination in step S212 is affirmative (step S212: Yes), the process proceeds to step S202. If the determination in step S212 is negative (step S212: No), the process proceeds to step S214. In step S214, the control unit 28 judges whether or not the process is to be terminated (step S214). The control unit 28 performs the judgment in step S214 by, for example, judging whether or not an end instruction has been input by the user operating the input unit 24A. If the determination in step S214 is negative (step S214: No), the process proceeds to step S210. If the determination in step S214 is affirmative (step S214: Yes), this routine ends.

[0112] As described above, the information processing system 1 of the present embodiment includes the first information processing device 10 connected to the photographing unit 30 that acquires the photographed image 40, and the second information processing device 20 that communicates with the first information processing device 10. The second information processing device 20 includes a receiving unit 28B, a converting unit 28C, and a display control unit 28D. The receiving unit 28B receives digitized data obtained by converting the photographed image 40 and the confirmation target image area T in which the confirmation target C included in the photographed image 40 is reflected, according to a conversion rule. The converting unit 28C converts the digitized data 42 into a situation image 44 that represents the situation of the confirmation target C reflected in the confirmation target image area T, based on the conversion rule. The display control unit 28D displays a superimposed image 46 in which the situation image 44 is superimposed on the photographed image 40.

[0113] Here, there are cases where the second information processing device 20 needs to remotely monitor the status of the confirmation target C located at a remote location in detail on the second information processing device 20 side. In the conventional technology, in order to confirm the status of the confirmation target C located at a remote location in detail, it was necessary to perform analysis using the captured image 40 received at a high image quality and a high frame rate. However, in the conventional technology, depending on the communication situation, degradation of image quality, low frame rate due to video delays and frame drops, etc. may occur, making it difficult to communicate high-image-quality video in real time, and therefore making it difficult to provide a highly accurate confirmation of the status of the confirmation target C.

[0114] On the other hand, in this embodiment, the receiving unit 28B of the second information processing device 20 receives from the first information processing device 10 the captured image 40 and digitized data obtained by converting the confirmation target image area T, which is included in the captured image 40 and contains the confirmation target C, in accordance with the conversion rule. The conversion unit 28C converts the digitized data 42 into a situation image 44 that represents the situation of the confirmation target C that is captured in the confirmation target image area T, based on the conversion rule. The display control unit 28D displays a superimposed image 46 in which the situation image 44 is superimposed on the captured image 40.

[0115] Since the digitized data 42 is data obtained by digitizing the confirmation target image area T according to the conversion rules, the amount of data is smaller than that of the photographed image 40, and it is less susceptible to the effects of communication conditions than the photographed image 40. This allows the receiving unit 28B to sequentially receive the digitized data 42 at substantially the same frame rate as when the digitized data 42 was transmitted from the transmitter and with the same values ​​as when it was transmitted. Then, the display control unit 28D converts the digitized data 42 into a situation image 44 that shows the situation of the confirmation target C captured in the confirmation target image area T based on the conversion rules, and displays a superimposed image 46 in which the situation image 44 is superimposed on the photographed image 40.

[0116] Therefore, even if the second information processing device 20 receives a captured image 40 with low image quality or a low frame rate due to communication conditions or the like, it is possible to display a superimposed image 46 superimposed with a situation image 44 showing the situation of the confirmation target C captured in the confirmation target image area T with high accuracy.

[0117] Therefore, by checking the situation image 44 of the superimposed image 46, the user can check the situation of the confirmation object C represented by the situation image 44 accurately and with a high degree of precision.

[0118] In addition, in the conventional technology, for example, when checking the blinking state of an LED, which is an example of a confirmation object C, at a remote location, a flicker phenomenon occurs in which the frame rate of the captured image 40 transmitted to a remote monitoring terminal is synchronized with the blinking speed of the LED, making it difficult for the monitoring terminal to provide an image that allows the status of the confirmation object C, such as the blinking pattern and blinking speed of the LED, to be understood. Also, when the captured image 40 captured at a high frame rate to avoid the flicker phenomenon is transmitted to the remote monitoring terminal at the same high frame rate, the amount of data is large and delays in the data transmission speed may occur.

[0119] Meanwhile, the receiving unit 28B of the second information processing device 20 of this embodiment receives, from the first information processing device 10, digitized data obtained by converting the captured image 40 and a confirmation target image area T, which is included in the captured image 40 and contains a confirmation target C, in accordance with a conversion rule. The conversion unit 28C converts the digitized data 42 into a situation image 44 representing the situation of the confirmation target C that is contained in the confirmation target image area T, based on the conversion rule. The display control unit 28D displays a superimposed image 46 in which the situation image 44 is superimposed on the captured image 40.

[0120] Therefore, the information processing system 1 of this embodiment can suppress the influence of the flicker phenomenon and provide the confirmation target C captured in the confirmation target image area T in a confirmable manner with accuracy and low transmission amount.

[0121] Therefore, the information processing system 1 of this embodiment can provide the status of the confirmation object C in a highly accurate manner so that it can be confirmed.

[0122] Furthermore, the receiving unit 28B of the second information processing device 20 of this embodiment receives the captured image 40 at a frame rate lower than that at which the captured image 40 was captured, and receives the digitized data 42 at a frame rate higher than that of the captured image 40. Then, the second information processing device 20 of this embodiment displays a superimposed image 46 in which situation images 44 generated based on each of the multiple pieces of digitized data 42 received sequentially at the high frame rate are superimposed on the captured image 40 received at the low frame rate.

[0123] Therefore, in addition to the above-mentioned effects, the information processing system 1 of the present embodiment can provide accurate and real-time confirmation of the status of the confirmation target C captured in the confirmation target image area T. Specifically, the information processing system 1 can provide, for example, confirmation of how the confirmation target C moves and how the status is changing in real time.

[0124] Next, an example of the hardware configuration of the first information processing device 10 and the second information processing device 20 of the above embodiment will be described.

[0125] FIG. 7 is a diagram showing an example of a hardware configuration of the first information processing device 10 and the second information processing device 20 of the above embodiment.

[0126] The first information processing device 10 and the second information processing device 20 of the above embodiment have a hardware configuration utilizing a normal computer, in which a CPU (Central Processing Unit) 11A, a ROM (Read Only Memory) 11B, a RAM (Random Access Memory) 11C, an I / F 11D, etc. are interconnected via a bus 11E.

[0127] The CPU 11A is a calculation device that controls the first information processing device 10 and the second information processing device 20 of the above embodiment. The ROM 11B stores programs and the like that realize various processes by the CPU 11A. Although a CPU is used for the explanation here, a GPU (Graphics Processing Unit) may be used as the calculation device that controls the first information processing device 10 and the second information processing device 20. The RAM 11C stores data necessary for various processes by the CPU 11A. The I / F 11D is an interface for transmitting and receiving data.

[0128] In the first information processing device 10 and the second information processing device 20 of the above-mentioned embodiments, the CPU 111A reads out a program from the ROM 11B onto the RAM 11C and executes it, thereby realizing each of the above-mentioned functions on the computer.

[0129] The programs for executing the above-mentioned processes executed by the first information processing device 10 and the second information processing device 20 of the above-mentioned embodiment may be stored in a HDD (hard disk drive). Also, the programs for executing the above-mentioned processes executed by the first information processing device 10 and the second information processing device 20 of the above-mentioned embodiment may be provided by being pre-installed in the ROM 11B.

[0130] The programs for executing the above-mentioned processes executed by the first information processing device 10 and the second information processing device 20 of the above-mentioned embodiment may be stored in a computer-readable storage medium such as a CD-ROM, a CD-R, a memory card, a DVD (Digital Versatile Disc), or a flexible disk (FD) in an installable or executable format file and provided as a computer program product. The programs for executing the above-mentioned processes executed by the first information processing device 10 and the second information processing device 20 of the above-mentioned embodiment may be stored on a computer connected to a network such as the Internet and provided by downloading the programs via the network. The programs for executing the above-mentioned processes executed by the first information processing device 10 and the second information processing device 20 of the above-mentioned embodiment may be provided or distributed via a network such as the Internet.

[0131] Although the embodiment of the present invention has been described above, this embodiment is presented as an example and is not intended to limit the scope of the invention. This new embodiment can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. This embodiment and its modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0132] 1. Information Processing Systems 10 First information processing device 18B Acquisition Department 18C generator 18D Communication control section 28B Receiver 28C Converter 28D Display control unit

Claims

1. An information processing system including a first information processing device connected to an image capturing unit that captures an image, and a second information processing device that communicates with the first information processing device, The second information processing device is a receiving unit for receiving the photographed image and digitalized data obtained by converting average values ​​of pixel values ​​constituting a confirmation target image area, which is included in the photographed image, in accordance with a conversion rule; a conversion unit that converts the digitized data converted for each captured image into a situation image that represents a situation of the confirmation target captured in the confirmation target image area based on the conversion rule; a display control unit that displays a superimposed image in which the situation image is superimposed on the captured image; Equipped with The conversion rule is: a threshold value for binarization, and status information of the confirmation target image region represented by each of the binarized values ​​of the average values ​​of the pixel values ​​constituting the confirmation target image region; The digitized data is the average value of the pixel values ​​constituting the confirmation target image region is binarized using the threshold value represented by the conversion rule; The object to be checked is a flashing light, The status information indicates whether the light is off or on. Information processing system.

2. The receiving unit receiving the captured image at a frame rate lower than that at which the captured image was captured; receiving the digitized data at a frame rate higher than that of the captured image; The information processing system according to claim 1 .

3. The confirmation target image area is The captured image includes an area in which the confirmation target is captured and which is an observation target for at least two types of situation changes. The information processing system according to claim 1 .

4. an acquisition unit that acquires a photographed image from the photographing unit; a generating unit that generates digitized data by converting an average value of pixel values ​​constituting a confirmation target image area in which a confirmation target is captured, the confirmation target being included in the photographed image, in accordance with a conversion rule; a communication control unit for transmitting the captured image and the digitalized data generated for each frame of the captured image; Equipped with The conversion rule is: a threshold value for binarization, and status information of the confirmation target image region represented by each of the binarized values ​​of the average values ​​of the pixel values ​​constituting the confirmation target image region; The digitized data is the average value of the pixel values ​​constituting the confirmation target image region is binarized using the threshold value represented by the conversion rule; The object to be checked is a flashing light, The status information indicates whether the light is off or on. Information processing device.

5. a receiving unit that receives a photographed image and digitalized data obtained by converting average values ​​of pixel values ​​constituting a confirmation target image area, which is included in the photographed image, in accordance with a conversion rule; a conversion unit that converts the digitized data converted for each captured image into a situation image that represents a situation of the confirmation target captured in the confirmation target image area based on the conversion rule; a display control unit that displays a superimposed image in which the situation image is superimposed on the captured image; Equipped with The conversion rule is: a threshold value for binarization, and status information of the confirmation target image region represented by each of the binarized values ​​of the average values ​​of the pixel values ​​constituting the confirmation target image region; The digitized data is the average value of the pixel values ​​constituting the confirmation target image region is binarized using the threshold value represented by the conversion rule; The object to be checked is a flashing light, The status information indicates whether the light is off or on. Information processing device.

6. An information processing program for causing a computer to execute the program, acquiring a photographed image from an imaging unit; generating digitalized data by converting an average value of pixel values ​​constituting a confirmation target image area, which is included in the photographed image and in which the confirmation target is captured, according to a conversion rule; transmitting the captured image and the digitalized data generated for each frame of the captured image; Including, The conversion rule is: a threshold value for binarization, and status information of the confirmation target image region represented by each of the binarized values ​​of the average values ​​of the pixel values ​​constituting the confirmation target image region; The digitized data is the average value of the pixel values ​​constituting the confirmation target image region is binarized using the threshold value represented by the conversion rule; The object to be checked is a flashing light, The status information indicates whether the light is off or on. Information processing program.

7. An information processing program for causing a computer to execute the program, receiving a photographed image and digitalized data obtained by converting average values ​​of pixel values ​​constituting a confirmation target image area, which is included in the photographed image and includes a confirmation target, according to a conversion rule; converting the digitized data converted for each captured image into a situation image representing a situation of the confirmation target captured in the confirmation target image area based on the conversion rule; displaying a superimposed image in which the situation image is superimposed on the captured image; Including, The conversion rule is: a threshold value for binarization, and status information of the confirmation target image region represented by each of the binarized values ​​of the average values ​​of the pixel values ​​constituting the confirmation target image region; The digitized data is the average value of the pixel values ​​constituting the confirmation target image region is binarized using the threshold value represented by the conversion rule; The object to be checked is a flashing light, The status information indicates whether the light is off or on. Information processing program.

Images