Device operating apparatus and device operating method

Abstract

A device operation device (3) is configured to have: a captured image acquisition unit (11) that acquires, from a camera (2), image data indicating a captured image in which a device (1) to be operated and a peripheral region of the device (1) to be operated are reflected; and a difference detection unit (12) that performs detection processing of a difference between the captured image indicated by the image data acquired by the captured image acquisition unit (11) and the registered image that has been recorded. Furthermore, the device operation device (3) has an image recording unit (14) that records the captured image as a registered image if the difference detection unit (12) detects the difference, acquires a second captured image that differs from the registered image that has been recorded if the difference detection unit (12) does not detect the difference, and records the captured image as the registered image if the difference detection unit (12) does not detect the difference. Recording the second captured image as a registered image; and a downloading unit (15) that downloads an operation application for the device (1) to be operated, which is shown in the registered image recorded by the image recording unit (14).

Description

Technical Field

[0001] The present disclosure relates to a device operating apparatus and a device operating method. Background Art

[0002] There is a device operating apparatus that accepts an operation of an operation target device by executing an operation application of the operation target device (see, for example, Patent Document 1).

[0003] The device operation apparatus comprises a first peripheral image acquisition unit, a second peripheral image acquisition unit, a setting unit and a signal output unit. The first peripheral image acquisition unit acquires an image of the peripheral area of ​​each of more than one operation object device, namely, a first peripheral image, and stores each first peripheral image in association with the operation object device. The second peripheral image acquisition unit acquires an image of the peripheral area of ​​the operation object device, namely, a second peripheral image, which is captured when the user operates the operation object device. If there is a first peripheral image that is consistent with the second peripheral image among multiple first peripheral images, the setting unit sets the operation object device corresponding to the first peripheral image as the device actually operated. If there is no first peripheral image that is consistent with the second peripheral image among multiple first peripheral images, the setting unit accepts the user's selection operation for the device actually operated. The signal output unit executes an operation application for the device actually operated, thereby outputting an operation signal to the device.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2012-119935 Summary of the invention

[0007] Problems to be solved by the invention

[0008] For example, in a private house, there are sometimes multiple rooms of the same shape and the same pattern of the ceiling, floor or wall. In multiple rooms with the same pattern of the wall, for example, the same model of air conditioner is sometimes installed as the operation target device. In this case, the surrounding areas of the air conditioners installed in the multiple rooms are similar to each other, so it is sometimes difficult to distinguish the images of the surrounding areas of the air conditioners installed in each room.

[0009] In the device operating apparatus disclosed in patent document 1, the first peripheral images of the operating object devices respectively arranged in a plurality of rooms are similar to each other. Therefore, a plurality of first peripheral images similar to each other are sometimes associated with the operating object devices and stored respectively by the first peripheral image acquisition unit. In this case, the setting unit sometimes fails to detect a first peripheral image consistent with the second peripheral image among the plurality of first peripheral images, or mistakenly detects a first peripheral image inconsistent with the second peripheral image. In addition, the user sometimes encounters the following situation: it is difficult to determine which operating object device is the device actually operated among the operating object devices associated with each first peripheral image. Under these circumstances, there is a problem that the signal output unit sometimes fails to execute the operating application of the device actually operated.

[0010] The present disclosure is completed to solve the above-mentioned problems, and its purpose is to obtain the following device operation device and device operation method: even if the captured image and the recorded registration image are similar and no difference between the two images is detected, an application for operating the operation object device reflected in the captured image can be executed.

[0011] Means for solving problems

[0012] The device operation device disclosed in the present invention comprises: a captured image acquisition unit that acquires image data representing a captured image showing an operation target device and a surrounding area of ​​the operation target device from a camera; and a difference detection unit that performs a detection process of a difference between the captured image represented by the image data acquired by the captured image acquisition unit and a recorded registration image. In addition, the device operation device comprises: an image recording unit that records the captured image as a registration image if the difference detection unit detects a difference, and acquires a second captured image that is different from the recorded registration image and records the second captured image as a registration image if the difference detection unit does not detect a difference; and a download unit that downloads an application for operating the operation target device shown in the registration image recorded by the image recording unit.

[0013] Effects of the Invention

[0014] According to the present disclosure, even when the captured image and the recorded registered image are similar and no difference between the two images is detected, the application for operating the operation target device reflected in the captured image can be executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] [ Figure 1 ] is a structural diagram showing the equipment operating device 3 of embodiment 1.

[0016] [ Figure 2 ] is a hardware structure diagram showing the hardware of the device operating device 3 according to embodiment 1.

[0017] [ Figure 3 ] is a hardware structure diagram of a computer when the device operating device 3 is implemented by software or firmware, etc.

[0018] [ Figure 4 ] is a flowchart showing the processing steps of the device operating device 3, that is, the device operating method.

[0019] [ Figure 5 ] is an explanatory diagram showing a captured image showing the operation target device 1 and the surrounding area of ​​the operation target device 1.

[0020] [ Figure 6 ] is an explanatory diagram showing an example of an identification mark.

[0021] [ Figure 7 ] is an explanatory diagram showing a display example of one or more operating applications downloaded by the download unit 15.

[0022] [ Figure 8 ] is a structural diagram showing the equipment operating device 3 of embodiment 2.

[0023] [ Fig. 9 ] is a hardware structure diagram showing the hardware of the device operating device 3 of embodiment 2.

[0024] [ Fig.10 ] is a structural diagram showing the equipment operating device 3 of embodiment 3.

[0025] [ Fig.11 ] is a hardware structure diagram showing the hardware of the device operating device 3 of embodiment 3.

[0026] [ Fig.12 ] Fig.12 A is an explanatory diagram showing an example of a captured image PG to which no grid is added. Fig.12 B is a diagram showing a captured image PG with a grid added. G An illustration of an example of .

[0027] [ Fig.13 ] is a structural diagram showing the equipment operating device 3 of embodiment 4.

[0028] [ Fig.14 ] is a hardware structure diagram showing the hardware of the device operating device 3 of embodiment 4.

[0029] [ Fig.15 ] is a diagram showing the characteristic region FR m An illustration of an example of . DETAILED DESCRIPTION

[0030] Hereinafter, in order to explain the present disclosure in more detail, the mode for implementing the present disclosure will be described with reference to the drawings.

[0031] Implementation Method 1

[0032] Figure 1 This is a diagram showing the configuration of the device operating device 3 according to the first embodiment.

[0033] Figure 2 1 is a hardware configuration diagram showing the hardware of the device operating device 3 according to the first embodiment.

[0034] exist Figure 1 In the embodiment, the operation target device 1 is, for example, an air conditioner, a television, or an audio device. Here, for the sake of convenience, the operation target device 1 is described as an air conditioner.

[0035] The camera 2 captures an image of a range including the operation target device 1 and the surrounding area of ​​the operation target device 1, and outputs image data representing a captured image PG to the device operating apparatus 3. The captured image PG may be a still image or a moving image.

[0036] exist Figure 1 In the device operating apparatus 3 shown, the camera 2 is provided outside the device operating apparatus 3. However, this is only an example, and the camera 2 may be built in the device operating apparatus 3.

[0037] The device operating apparatus 3 includes a captured image acquiring unit 11 , a difference detecting unit 12 , a guidance presenting unit 13 , an image recording unit 14 , a downloading unit 15 , and an operation accepting unit 16 .

[0038] The captured image acquisition unit 11 performs, for example, Figure 2 It is implemented by the captured image acquisition circuit 31 shown.

[0039] The captured image acquisition unit 11 acquires image data representing the captured image PG from the camera 2 .

[0040] If the difference detection unit 12 does not detect that the captured image PG is different from the registered images RG1 to RG N , the captured image acquisition unit 11 obtains image data representing a captured image of the operation target device 1 with the identification mark attached thereto and the surrounding area of ​​the operation target device 1 from the camera 2 as the second captured image PG' after being prompted by the guidance prompting unit 13. N is an integer greater than 1.

[0041] The captured image acquisition unit 11 outputs image data indicating the captured image PG to each of the difference detection unit 12 and the image recording unit 14 .

[0042] The captured image acquisition unit 11 outputs image data indicating the second captured image PG′ to the image recording unit 14 .

[0043] The difference detection unit 12 performs, for example, Figure 2 The difference detection circuit 32 is shown to achieve this.

[0044] The difference detection unit 12 obtains image data indicating the captured image PG from the captured image acquisition unit 11 and obtains data indicating the recorded registered image RG from the image recording unit 14. n (n=1, ..., N) image data.

[0045] The difference detection unit 12 compares the captured image PG with the recorded registered image RG. n The difference detection process is to detect the difference between the captured image PG and the recorded registration image RG. n A comparison is performed to determine whether there is any difference in the processing of the two images.

[0046] The guidance and presentation unit 13 may, for example, Figure 2 The guidance prompt circuit 33 shown is implemented.

[0047] If the difference detecting unit 12 does not detect a difference, the guidance presenting unit 13 presents guidance for prompting the user to attach the identification mark.

[0048] As a method of presenting the guidance, the guidance presenting unit 13 may display the guidance on a display (not shown), or the guidance may be output in the form of voice from a speaker (not shown).

[0049] The image recording unit 14 is configured to store the image data in a predetermined manner. Figure 2 This is achieved by the image recording circuit 34 shown.

[0050] When the difference detecting unit 12 detects a difference, the image recording unit 14 acquires image data representing the captured image PG from the captured image acquiring unit 11 .

[0051] The image recording unit 14 records the captured image PG as the registration image RG n+1 (n=1,…,N).

[0052] If the difference detection unit 12 does not detect a difference, the image recording unit 14 obtains from the captured image acquisition unit 11 an image indicating that the image is different from the registered image RG that has been recorded. n There is image data of a differentiated second captured image PG'.

[0053] The image recording unit 14 records the second captured image PG′ as the registration image RG n+1 .

[0054] That is, if the difference detecting unit 12 detects no difference, the image recording unit 14 records the captured image of the operation target device 1 to which the identification mark is attached and the surrounding area of ​​the operation target device 1 as the second captured image PG′ as the registration image RG. n+1 .

[0055] The downloading unit 15 is for example Figure 2 The download circuit 35 shown is used to implement this.

[0056] The downloading unit 15 downloads the registration image RG recorded by the image recording unit 14 from, for example, the server device 4 of the manufacturer of the operation target device 1 via the network. n+1 An application for operating the operation target device 1 shown in FIG.

[0057] The operating application of the operation target device 1 is stored in, for example, the internal memory of the download unit 15 .

[0058] The operation receiving unit 16 performs, for example, Figure 2 The operation acceptance circuit 36 ​​shown is implemented.

[0059] The operation accepting unit 16 accepts selection of any one operating application from among the one or more operating applications downloaded by the downloading unit 15 .

[0060] The operation accepting unit 16 executes the operating application selected by the user, thereby accepting an operation of the operation target device 1 .

[0061] exist Figure 1 In the embodiment, it is assumed that the captured image acquisition unit 11, the difference detection unit 12, the guidance prompt unit 13, the image recording unit 14, the downloading unit 15 and the operation receiving unit 16 as the structural elements of the device operation device 3 are respectively Figure 2 That is, it is assumed that the device operating device 3 is implemented by a captured image acquisition circuit 31, a difference detection circuit 32, a guidance display circuit 33, an image recording circuit 34, a download circuit 35, and an operation acceptance circuit 36.

[0062] The image acquisition circuit 31, the difference detection circuit 32, the guidance prompt circuit 33, the image recording circuit 34, the download circuit 35 and the operation acceptance circuit 36 ​​are, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or a combination of these components.

[0063] The components of the device operating apparatus 3 are not limited to those realized by dedicated hardware, and the device operating apparatus 3 may be realized by software, firmware, or a combination of software and firmware.

[0064] Software or firmware is stored as a program in a computer's memory. Computer refers to hardware that executes programs, such as CPU (Central Processing Unit), central processing unit, processing unit, computing unit, microprocessor, microcomputer, processor or DSP (Digital Signal Processor).

[0065] Figure 3 This is a hardware configuration diagram of a computer when the device operating device 3 is implemented by software, firmware, or the like.

[0066] When the device operating device 3 is implemented by software or firmware, a program for causing the computer to execute the processing steps of the captured image acquisition unit 11, the difference detection unit 12, the guidance prompt unit 13, the image recording unit 14, the download unit 15, and the operation acceptance unit 16 is stored in the memory 51. Then, the processor 52 of the computer executes the program stored in the memory 51.

[0067] In addition, Figure 2 In FIG. 1 , an example is shown in which the components of the device operating device 3 are respectively implemented by dedicated hardware. Figure 3 , an example is shown in which the device operating device 3 is implemented by software or firmware. However, this is just an example, and some components of the device operating device 3 may be implemented by dedicated hardware, and the remaining components may be implemented by software or firmware.

[0068] Next, Figure 1 The operation of the device operating device 3 shown will be described.

[0069] Figure 4 It is a flowchart showing the processing procedure of the device operating apparatus 3, that is, the device operating method.

[0070] The user operates camera 2, such as Figure 5 As shown, a range including the operation target device 1 and the surrounding area of ​​the operation target device 1 is captured.

[0071] Figure 5 1 is an explanatory diagram showing a captured image showing the operation target device 1 and the surrounding area of ​​the operation target device 1 .

[0072] exist Figure 5In the example of , the surrounding area of ​​the operation target device 1 includes the wall on which the operation target device 1 is installed, the curtain installed near the operation target device 1, and the ceiling near the wall on which the operation target device 1 is installed.

[0073] exist Figure 5 In the example of FIG. 2 , when the operation target device 1 is recognized by the camera 2 , a focus mark is displayed in the viewfinder of the camera 2 .

[0074] When the user clicks the operation target device 1 recognized by the camera 2 , the operation target device 1 is determined.

[0075] The camera 2 captures an image of a range including the operation target device 1 and the surrounding area of ​​the operation target device 1 , and outputs image data representing the captured image PG to the device operating apparatus 3 .

[0076] The captured image acquisition unit 11 acquires image data representing the captured image PG from the camera 2 ( Figure 4 Step ST1).

[0077] The captured image acquisition unit 11 outputs image data indicating the captured image PG to each of the difference detection unit 12 and the image recording unit 14 .

[0078] The difference detection unit 12 acquires image data indicating the captured image PG from the captured image acquisition unit 11 and acquires data indicating the recorded registered images RG1 to RG2 from the image recording unit 14. N image data.

[0079] The difference detection unit 12 compares the captured image PG with the recorded registration image RG. n (n=1, ..., N) by comparing the captured image PG with the recorded registration image RG. n The difference between the detection process ( Figure 4 Step ST2).

[0080] As a difference detection process, a known image comparison application that determines whether a plurality of images are the same image can be used. As a known image comparison application, there is an application using a pattern matching method, for example.

[0081] Specifically, the difference detection unit 12 determines whether the captured image PG and the recorded registered image RG n If the similarity is greater than the threshold, the difference detection unit 12 performs a n If the similarity is less than the threshold, the difference detection unit 12 performs a comparison between the captured image PG and the registered image RG. nThe threshold value for similarity determination may be stored in the internal memory of the difference detection unit 12 or provided from outside the device operating device 3.

[0082] The difference detection unit 12 outputs the difference detection processing result to the captured image acquisition unit 11 , the guidance presentation unit 13 , and the image recording unit 14 .

[0083] The image recording unit 14 acquires the difference detection processing result from the difference detecting unit 12 .

[0084] If the difference detection unit 12 detects a difference ( Figure 4 Step ST3: If yes), the image recording unit 14 records the captured image PG as the registration image RG n+1 ( Figure 4 Step ST4).

[0085] The guidance presenting unit 13 obtains the difference detection processing result from the difference detecting unit 12 .

[0086] If the difference detection unit 12 does not detect a difference ( Figure 4 Step ST3: If no), the guidance prompting unit 13 prompts the user to attach the identification mark ( Figure 4 Step ST5).

[0087] like Figure 6 As shown, the user who has received the guidance prompt performs an operation of attaching the identification mark to the operation target device 1 .

[0088] Figure 6 This is an explanatory diagram showing an example of an identification mark.

[0089] The identification mark may be a mark of any pattern. The identification mark may be packaged together with the corrugated cardboard that packages the purchased operation target device 1, for example.

[0090] The captured image PG is already registered as the image RG. n If recorded, the user who has received the guidance prompt does not perform the operation of attaching the identification mark to the operation target device 1 to prevent double registration of the captured image PG.

[0091] When the user performs an operation to affix the identification mark to the operation target device 1 , the user operates the camera 2 to capture an image of the range including the operation target device 1 to which the identification mark is affixed and the surrounding area of ​​the operation target device 1 .

[0092] The camera 2 captures an image of the operation target device 1 to which the identification mark is attached and the surrounding area of ​​the operation target device 1 as a second captured image PG′, and outputs image data representing the second captured image PG′ to the device operating apparatus 3 .

[0093] The captured image acquisition unit 11 acquires the difference detection processing result from the difference detection unit 12 .

[0094] If the difference detection unit 12 does not detect a difference, the captured image acquisition unit 11 acquires image data (i) representing the second captured image PG′ output from the camera 2 after the guidance is prompted by the guidance prompting unit 13. Figure 4 Step ST6).

[0095] The captured image acquisition unit 11 outputs image data indicating the second captured image PG′ to the image recording unit 14 .

[0096] If the difference detection unit 12 does not detect a difference, the image recording unit 14 acquires image data representing the second captured image PG′ from the captured image acquisition unit 11 .

[0097] The image recording unit 14 records the second captured image PG′ as the registration image RG n+1 ( Figure 4 Step ST7).

[0098] When the image recording unit 14 records the captured image PG or the second captured image PG′ as the registration image RG n+1 Then, for example, the downloading unit 15 downloads the registration image RG from the server device 4 of the manufacturer of the operation target device 1. n+1 The operation application for the operation target device 1 shown in the image ( Figure 4 Step ST8).

[0099] The operating application of the operation target device 1 is stored in, for example, the internal memory of the download unit 15 .

[0100] When the user operates the operation target device 1, Figure 7 As shown, the operation accepting unit 16 displays one or more operation applications downloaded by the downloading unit 15 on a display (not shown). At this time, the operation accepting unit 16 displays the registration image RG corresponding to each operation application on the display. n+1 (n=1,…,N).

[0101] Figure 7 It is an explanatory diagram showing a display example of one or more operating applications downloaded by the downloading unit 15 .

[0102] exist Figure 7In the example of FIG. 1 , as the operating applications of the operation target device 1 , an operating application for the air conditioner A, an operating application for the air conditioner B, an operating application for the television A, and an operating application for the television B are displayed respectively.

[0103] In addition, Figure 7 In the example, registration image RG1 corresponding to the operating application of air conditioner A, registration image RG2 corresponding to the operating application of air conditioner A, registration image RG3 corresponding to the operating application of television A, and registration image RG3 corresponding to the operating application of television B are displayed respectively.

[0104] The operation accepting unit 16 accepts selection of any one operation application from one or more selectable operation applications ( Figure 4 Step ST9).

[0105] exist Figure 7 In the example of FIG. 1 , an application for operating the air conditioner B as the operation target device 1 is selected. The header of the air conditioner B is selected.

[0106] The operation accepting unit 16 executes the operation application selected by the user, thereby accepting the operation of the operation target device 1 ( Figure 4 Step ST10).

[0107] In the above-mentioned first embodiment, the device operation apparatus 3 is configured to include: a captured image acquisition unit 11 that acquires image data representing a captured image in which the operation target device 1 and the surrounding area of ​​the operation target device 1 are reflected from the camera 2; and a difference detection unit 12 that performs a detection process of a difference between the captured image represented by the image data acquired by the captured image acquisition unit 11 and a recorded registration image. In addition, the device operation apparatus 3 includes: an image recording unit 14 that records the captured image as a registration image if the difference detection unit 12 detects a difference, and acquires a second captured image that is different from the recorded registration image and records the second captured image as a registration image if the difference detection unit 12 does not detect a difference; and a downloading unit 15 that downloads an operation application for the operation target device 1 reflected in the registration image recorded by the image recording unit 14. Therefore, even when the captured image and the recorded registration image are similar and no difference between the images is detected, the device operation apparatus 3 can execute the operation application for the operation target device reflected in the captured image.

[0108] exist Figure 1 In the device operating apparatus 3 shown in the figure, the captured image acquisition unit 11 acquires image data representing one captured image PG, and the difference detection unit 12 compares the one captured image PG with the recorded registered image RG. nHowever, this is just an example, and the captured image acquisition unit 11 may acquire image data representing a plurality of captured images PG, and the difference detection unit 12 may perform a comparison between the plurality of captured images PG and the registered image RG. n The difference between the detection process.

[0109] The plurality of captured images PG are, for example, images obtained by capturing the same operation target device 1 from different directions.

[0110] The difference detection unit 12 compares the plurality of captured images PG with the recorded registered images RG. n The difference detection process between the captured image PG and the recorded registered image RG is performed. n Compared with the case of the difference detection processing, the difference detection accuracy is improved.

[0111] exist Figure 1 In the device operating apparatus 3 shown, the captured image acquisition unit 11 acquires image data representing a captured image showing the device to be operated 1 and the surrounding area of ​​the device to be operated 1. For example, before the captured image acquisition unit 11 acquires the image data, in the case where the device to be operated 1 and the surrounding area are associated, the captured image acquisition unit 11 may acquire image data representing a captured image showing only the surrounding area of ​​the device to be operated 1.

[0112] Implementation Method 2

[0113] In embodiment 2, the following device operating apparatus 3 is described: if the difference detection unit 12 detects no difference, the image recording unit 17 synthesizes an illustration component representing an identification mark for the operation object device 1 projected in the captured image PG, thereby obtaining an image of the operation object device 1 with the identification mark and the surrounding area as the second captured image.

[0114] Figure 8 2 is a diagram showing the structure of the device operating device 3 according to the second embodiment. Figure 8 In, with Figure 1 The same reference numerals denote the same or corresponding parts, and thus their descriptions are omitted.

[0115] Fig. 9 2 is a hardware configuration diagram showing the hardware of the device operating device 3 according to the second embodiment. Fig. 9 In, with Figure 2 The same reference numerals denote the same or corresponding parts, and thus their descriptions are omitted.

[0116] Figure 8 The device operating apparatus 3 shown includes a captured image acquiring unit 11 , a difference detecting unit 12 , an image recording unit 17 , a downloading unit 15 , and an operation accepting unit 16 . Figure 8 The equipment operating device 3 shown is Figure 1 The device operating device 3 shown is different and does not include the guidance presenting unit 13 .

[0117] The image recording unit 17 is configured to store the image of the Fig. 9 This is achieved by the image recording circuit 37 shown.

[0118] If the difference detection unit 12 detects a difference, the image recording unit 17 records the captured image PG as the registration image RG. n+1 .

[0119] If the difference detecting unit 12 detects no difference, the image recording unit 17 synthesizes an illustration component representing the identification mark with respect to the operation target device 1 shown in the captured image PG, thereby acquiring an image showing the operation target device 1 with the identification mark and the surrounding area as a second captured image PG′.

[0120] The image recording unit 17 records the second captured image PG′ as the registration image RG n+1 .

[0121] exist Figure 8 In the embodiment, it is assumed that the captured image acquisition unit 11, the difference detection unit 12, the image recording unit 17, the downloading unit 15 and the operation receiving unit 16 as the structural elements of the device operation device 3 are respectively Fig. 9 That is, it is assumed that the device operating device 3 is implemented by a captured image acquisition circuit 31, a difference detection circuit 32, an image recording circuit 37, a download circuit 35, and an operation acceptance circuit 36.

[0122] The image acquisition circuit 31, the difference detection circuit 32, the image recording circuit 37, the download circuit 35 and the operation acceptance circuit 36 ​​are, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC, an FPGA or a combination thereof.

[0123] The components of the device operating apparatus 3 are not limited to those realized by dedicated hardware, and the device operating apparatus 3 may be realized by software, firmware, or a combination of software and firmware.

[0124] When the device operating device 3 is implemented by software or firmware, a program for causing a computer to execute the processing steps of the captured image acquisition unit 11, the difference detection unit 12, the image recording unit 17, the downloading unit 15, and the operation receiving unit 16 is stored in Figure 3 The memory 51 shown. Moreover, Figure 3 The illustrated processor 52 executes programs stored in the memory 51 .

[0125] In addition, Fig. 9In FIG. 1 , an example is shown in which the components of the device operating device 3 are respectively implemented by dedicated hardware. Figure 3 , an example is shown in which the device operating device 3 is implemented by software or firmware. However, this is just an example, and some components of the device operating device 3 may be implemented by dedicated hardware, and the remaining components may be implemented by software or firmware.

[0126] Next, Figure 8 The operation of the device operating device 3 shown in FIG. 1 is described below. Figure 1 The device operating device 3 shown is the same. Therefore, only the operation of the image recording unit 17 will be described here.

[0127] If the difference detection unit 12 detects a difference, Figure 1 Similarly to the image recording unit 14 shown in FIG. 1 , the image recording unit 17 records the captured image PG as the registration image RG. n+1 .

[0128] If the difference detection unit 12 does not detect a difference, the image recording unit 17 synthesizes an illustration component representing an identification mark with respect to the operation target device 1 projected in the captured image PG. The illustration component representing the identification mark may be stored in the internal memory of the image recording unit 17, or may be provided from outside the device operating apparatus 3. Image processing for synthesizing the illustration component representing the identification mark with the operation target device 1 is a well-known technique, and therefore a detailed description thereof is omitted.

[0129] The image recording unit 17 records, as the second captured image PG′, an image of the operation target device 1 with the identification mark and the surrounding area as the registration image RG. n+1 .

[0130] In the above-mentioned second embodiment, the device operating apparatus 3 is configured such that, if the difference detecting unit 12 does not detect a difference, the image recording unit 17 synthesizes an illustration component representing the identification mark for the operation target device 1 reflected in the captured image, thereby obtaining an image reflecting the operation target device 1 with the identification mark and the surrounding area as the second captured image. Therefore, even if the captured image and the recorded registered image are similar and no difference is detected between the images of both parties, the device operating apparatus 3 can execute the operation application of the operation target device reflected in the captured image, and in addition, the user can omit the operation of attaching the identification mark to the operation target device 1.

[0131] Implementation 3

[0132] In the third embodiment, the device operating apparatus 3 including the grid adding unit 18 for adding a grid to the captured image PG or the second captured image PG′ will be described.

[0133] Fig.10 2 is a block diagram showing a device operating device 3 according to a third embodiment. Fig.10 In, with Figure 1 and Figure 8 The same reference numerals denote the same or corresponding parts, and thus their descriptions are omitted.

[0134] Fig.11 FIG. 2 is a hardware configuration diagram showing the hardware of the device operating device 3 according to the third embodiment. Fig.11 In, with Figure 2 and Fig. 9 The same reference numerals denote the same or corresponding parts, and thus their descriptions are omitted.

[0135] Fig.11 The device operating apparatus 3 shown includes a captured image acquiring unit 11 , a grid adding unit 18 , a difference detecting unit 12 , a guidance presenting unit 13 , an image recording unit 14 , a downloading unit 15 , and an operation accepting unit 16 .

[0136] The mesh adding section 18 is for example Fig.11 This is achieved by the grid addition circuit 38 shown.

[0137] The grid adding unit 18 adds a grid to the captured image PG upon acquiring the image data representing the captured image PG from the captured image acquiring unit 11 .

[0138] The grid adding unit 18 adds a grid to the captured image PG. G The images are output as the captured images PG to the difference detection unit 12 and the image recording unit 14 .

[0139] Upon acquiring the image data indicating the second captured image PG′ from the captured image acquisition unit 11 , the grid adding unit 18 adds a grid to the second captured image PG′.

[0140] The grid adding unit 18 adds a grid to the second captured image PG′. G The image is output to the image recording unit 14 as the second captured image PG.

[0141] exist Fig.10 In the device operating apparatus 3 shown, the grid attachment portion 18 is applied to Figure 1 The device operating device 3 shown. However, this is just an example, and the grid adding part 18 can also be applied to Figure 8 The device operating device 3 is shown.

[0142] exist Fig.10In the embodiment, it is assumed that the captured image acquisition unit 11, the grid adding unit 18, the difference detection unit 12, the guidance prompting unit 13, the image recording unit 14, the downloading unit 15 and the operation receiving unit 16 as the structural elements of the device operation device 3 are respectively Fig.11 That is, it is assumed that the device operating device 3 is implemented by a captured image acquisition circuit 31, a grid adding circuit 38, a difference detection circuit 32, a guidance prompting circuit 33, an image recording circuit 34, a downloading circuit 35, and an operation receiving circuit 36.

[0143] The captured image acquisition circuit 31, the grid addition circuit 38, the difference detection circuit 32, the guidance prompt circuit 33, the image recording circuit 34, the download circuit 35 and the operation acceptance circuit 36 ​​are, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC, an FPGA or a combination thereof.

[0144] The components of the device operating apparatus 3 are not limited to those realized by dedicated hardware, and the device operating apparatus 3 may be realized by software, firmware, or a combination of software and firmware.

[0145] When the device operating device 3 is implemented by software or firmware, a program for causing a computer to execute the processing steps of the captured image acquisition unit 11, the grid adding unit 18, the difference detection unit 12, the guidance prompting unit 13, the image recording unit 14, the downloading unit 15, and the operation receiving unit 16 is stored in Figure 3 The memory 51 shown. Moreover, Figure 3 The illustrated processor 52 executes programs stored in the memory 51 .

[0146] In addition, Fig.11 In FIG. 1 , an example is shown in which the components of the device operating device 3 are respectively implemented by dedicated hardware. Figure 3 , an example is shown in which the device operating device 3 is implemented by software or firmware. However, this is just an example, and some components of the device operating device 3 may be implemented by dedicated hardware, and the remaining components may be implemented by software or firmware.

[0147] Next, Fig.10 The operation of the device operating device 3 shown in FIG. 1 is described below. Figure 1 The device operating device 3 shown is the same. Therefore, here, the operation of the mesh adding unit 18 will be mainly described.

[0148] The grid adding unit 18 adds a grid to the photographed image PG upon acquiring image data representing the photographed image PG from the photographed image acquiring unit 11. The process of adding a grid to the photographed image PG is a well-known technique, and thus detailed description thereof will be omitted.

[0149] The grid adding unit 18 adds a grid to the captured image PG. G The images are output as the captured images PG to the difference detection unit 12 and the image recording unit 14 .

[0150] Upon acquiring the image data indicating the second captured image PG′ from the captured image acquisition unit 11 , the grid adding unit 18 adds a grid to the second captured image PG′.

[0151] The grid adding unit 18 adds a grid to the second captured image PG′. G The image is output to the image recording unit 14 as the second captured image PG.

[0152] Fig.12 The captured image PG with a grid is shown. G An illustration of an example of .

[0153] Fig.12 A shows an example of a captured image PG without a grid. Fig.12 B shows the captured image PG with a grid added G An example of.

[0154] exist Fig.12 In the example of , the intervals between the vertical lines and the horizontal lines constituting the grid are set to ○○ mm. As a specific value of ○○ mm, for example, 70 mm is used.

[0155] The difference detection unit 12 detects the grid-covered captured image PG G and the recorded registration image RG n (n=1, ..., N) to detect the captured image PG with a grid. G With the recorded registration image RG n The difference between the recorded registration image RG n A grid is also attached.

[0156] By adding a grid, for example, the size ratio between the operation target device 1 and objects existing in the surrounding area becomes clear. n The difference in detection accuracy between the captured images with grid PG G With the registration image RG with grid n The detection accuracy of the difference between them is higher.

[0157] If the difference detection unit 12 detects a difference, the image recording unit 14 records the captured image PG with a grid. G As the registration image RG n+1 .

[0158] If the difference detection unit 12 does not detect a difference, the image recording unit 14 records the captured image PG with a grid on which the identification mark is attached. G 'As the registration image RG n+1 .

[0159] In the above embodiment 3, Fig.10 The device operating apparatus 3 shown in the figure is configured to include a grid adding unit 18, which adds a grid to the captured image represented by the image data acquired by the captured image acquisition unit 11, and outputs the captured image with the grid to the difference detection unit 12 and the image recording unit 14, respectively. Therefore, when the captured image and the recorded registered image are similar and no difference between the two images is detected, Fig.10 The device operating apparatus 3 shown in the figure can also execute an application for operating the device to be operated shown in the captured image. Figure 1 Compared with the device operating device 3 shown in the figure, the detection accuracy of the difference between the captured image and the registered image can be improved.

[0160] Implementation 4

[0161] In the fourth embodiment, the device operating device 3 includes a feature region specification accepting unit 19, which performs a comparison between the captured image PG acquired by the captured image acquiring unit 11 and the recorded registered image RG by the difference detecting unit 12. n In the detection process of the difference between (n=1, . . . , N), the characteristic region specification accepting unit 19 accepts the specification of a characteristic region, which is a region to be noted in the captured image PG.

[0162] Fig.13 2 is a block diagram showing a device operating apparatus 3 according to a fourth embodiment. Fig.13 In, with Figure 1 , Figure 8 and Fig.10 The same reference numerals denote the same or corresponding parts, and thus their descriptions are omitted.

[0163] Fig.14 FIG. 2 is a hardware configuration diagram showing the hardware of the device operating device 3 according to the fourth embodiment. Fig.14 In, with Figure 2 , Fig. 9 and Fig.11 The same reference numerals denote the same or corresponding parts, and thus their descriptions are omitted.

[0164] Fig.14 The device operating apparatus 3 shown includes a captured image acquisition unit 11 , a characteristic region specification accepting unit 19 , a difference detecting unit 12 , a guidance presenting unit 13 , an image recording unit 14 , a downloading unit 15 , and an operation accepting unit 16 .

[0165] The characteristic region specification receiving unit 19 performs, for example, Fig.14 The characteristic region designation acceptance circuit 39 shown is implemented.

[0166] The difference detection unit 12 compares the captured image PG acquired by the captured image acquisition unit 11 with the registered image RG that has been recorded. n When detecting the difference between (n=1, . . . , N), the characteristic region designation accepting unit 19 accepts a characteristic region FR that is a region to be noted in the captured image PG. m m=1,…,M,where M is an integer greater than 1.

[0167] The characteristic region specification accepting unit 19 receives the specified characteristic region FR m PG F The images are output as the captured images PG to the difference detection unit 12 and the image recording unit 14 .

[0168] The characteristic region specification accepting unit 19 receives the specified characteristic region FR m PG F The image is output to the image recording unit 14 as the second captured image PG.

[0169] Furthermore, the characteristic region specification accepting unit 19 accepts the plurality of characteristic regions FR m When the specified feature area FR m Designation of priority.

[0170] The feature region designation accepting unit 19 receives the designated feature region FR with a priority order. m PG F The images are output as the captured images PG to the difference detection unit 12 and the image recording unit 14 .

[0171] The feature region designation accepting unit 19 receives the designated feature region FR with a priority order. m PG F The image is output to the image recording unit 14 as the second captured image PG.

[0172] exist Fig.13 In the device operating apparatus 3 shown in FIG. 1 , the characteristic region designation receiving unit 19 is used to Figure 1 However, this is only an example, and the characteristic region designation receiving unit 19 can also be applied to Figure 8The device operating device 3 shown or Fig.10 The device operating device 3 is shown.

[0173] exist Fig.13 In the embodiment, it is assumed that the captured image acquisition unit 11, the characteristic region designation acceptance unit 19, the difference detection unit 12, the guidance prompt unit 13, the image recording unit 14, the downloading unit 15 and the operation acceptance unit 16 as the structural elements of the device operation device 3 are respectively Fig.14 That is, it is assumed that the device operating device 3 is implemented by a captured image acquisition circuit 31, a feature area designation acceptance circuit 39, a difference detection circuit 32, a guidance prompt circuit 33, an image recording circuit 34, a download circuit 35, and an operation acceptance circuit 36.

[0174] The captured image acquisition circuit 31, the feature area designation acceptance circuit 39, the difference detection circuit 32, the guidance prompt circuit 33, the image recording circuit 34, the download circuit 35 and the operation acceptance circuit 36 ​​are, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC, an FPGA or a combination thereof.

[0175] The components of the device operating apparatus 3 are not limited to those realized by dedicated hardware, and the device operating apparatus 3 may be realized by software, firmware, or a combination of software and firmware.

[0176] When the device operating device 3 is implemented by software or firmware, a program for causing a computer to execute the processing steps of the captured image acquisition unit 11, the feature area designation acceptance unit 19, the difference detection unit 12, the guidance prompt unit 13, the image recording unit 14, the downloading unit 15, and the operation acceptance unit 16 is stored in Figure 3 The memory 51 shown. Moreover, Figure 3 The illustrated processor 52 executes programs stored in the memory 51 .

[0177] In addition, Fig.14 In FIG. 1 , an example is shown in which the components of the device operating device 3 are respectively implemented by dedicated hardware. Figure 3 , an example is shown in which the device operating device 3 is implemented by software or firmware. However, this is just an example, and some components of the device operating device 3 may be implemented by dedicated hardware, and the remaining components may be implemented by software or firmware.

[0178] Next, Fig.13 The operation of the device operating device 3 shown in FIG. 1 is described below. Figure 1 The device operating device 3 is the same as that shown in the figure. Therefore, here, the operation of the feature region specification accepting unit 19 will be mainly described.

[0179] The difference detection unit 12 compares the captured image PG with the recorded registered image RG. n When detecting the difference between (n=1, . . . , N), the characteristic region designation accepting unit 19 accepts the characteristic region FR that is the region to be noted. m (m=1, ..., M) is specified. Feature region FR specified by the user m The acceptance process itself is a well-known technology, so detailed description is omitted.

[0180] Fig.15 The characteristic region FR m An illustration of an example of .

[0181] exist Fig.15 In the example, as the feature region FR m , specifying the area FR1 for applying a heart-shaped mark to the curtain and the area FR2 for the left corner of the curtain.

[0182] Furthermore, the characteristic region specification accepting unit 19 accepts the plurality of characteristic regions FR m When the specified feature area FR m The acceptance process of the priority specified by the user is a well-known technology, so the detailed description is omitted.

[0183] exist Fig.15 In the example of , the characteristic region FR1 and the characteristic region FR2 are specified, and therefore the characteristic region specification accepting unit 19 accepts the specification of the priority order of the characteristic region FR1 and the characteristic region FR2.

[0184] Here, for convenience of explanation, it is assumed that the priority of the characteristic region FR1 is the first and the priority of the characteristic region FR2 is the second.

[0185] The feature region designation accepting unit 19 receives the designated feature region FR with a priority order. m PG F The images are output as the captured images PG to the difference detection unit 12 and the image recording unit 14 .

[0186] The feature region designation accepting unit 19 receives the designated feature region FR with a priority order. m PG F The image is output to the image recording unit 14 as the second captured image PG.

[0187] The difference detection unit 12 obtains the designated priority feature region FR from the feature region designation acceptance unit 19. m The captured image PG.

[0188] The difference detection unit 12 detects the captured image PG F The feature region FR1 with the first priority and the registered image RG n The region corresponding to the characteristic region FR1 is compared with the region (n=1, . . . , N), thereby detecting the difference between the characteristic region FR1 and the region corresponding to the characteristic region FR1.

[0189] If there is a difference between the characteristic region FR1 and the region corresponding to the characteristic region FR1 , the difference detecting unit 12 ends the matching process.

[0190] If there is no difference between the feature region FR1 and the region corresponding to the feature region FR1, the difference detecting unit 12 compares the feature region FR2 with the second priority and the registered image RG. n The region corresponding to the characteristic region FR2 is compared thereto, thereby detecting the difference between the characteristic region FR2 and the region corresponding to the characteristic region FR2.

[0191] If the difference detection unit 12 detects a difference, the image recording unit 14 records the image PG in which the characteristic region FR is specified. m PG F As the registration image RG n+1 .

[0192] If the difference detection unit 12 does not detect a difference, the image recording unit 14 records the feature region FR assigned with a priority as the second captured image PG′. m PG F 'As the registration image RG n+1 .

[0193] In the above embodiment 4, Fig.13 The device operating device 3 shown is configured to include a feature area designation accepting unit 19. When the difference detection unit 12 performs a detection process on the difference between the captured image acquired by the captured image acquisition unit 11 and the recorded registered image, the feature area designation accepting unit 19 accepts the designation of a feature area that should be paid attention to in the captured image, and outputs the captured image to which the feature area with a priority order is designated to the difference detection unit 12 and the image recording unit 14, respectively. Therefore, in the case where the captured image and the recorded registered image are similar and no difference is detected between the two images, Fig.13 The device operating apparatus 3 shown in the figure can also execute an application for operating the device to be operated shown in the captured image. Figure 1 Compared with the device operating apparatus 3 shown in the figure, the processing load of the difference detection process in the difference detecting unit 12 can be reduced.

[0194] In addition, the present disclosure allows free combination of the various embodiments, modification of arbitrary components of the various embodiments, or omission of arbitrary components of the various embodiments.

[0195] Industrial Applicability

[0196] The present disclosure is applicable to a device operating apparatus and a device operating method.

[0197] Description of symbols

[0198] 1: Operation object device; 2: Camera; 3: Device operating device; 4: Server device; 11: Image capturing and acquiring unit; 12: Difference detecting unit; 13: Guidance prompting unit; 14: Image recording unit; 15: Downloading unit; 16: Operation accepting unit; 17: Image recording unit; 18: Grid attaching unit; 19: Feature area designation accepting unit; 31: Image capturing and acquiring circuit; 32: Difference detecting circuit; 33: Guidance prompting circuit; 34: Image recording circuit; 35: Downloading circuit; 36: Operation accepting circuit; 37: Image recording circuit; 38: Grid attaching circuit; 39: Feature area designation accepting circuit; 51: Memory; 52: Processor.

Claims

1. A device operating device, the device operating device having: a captured image acquisition unit that acquires, from the camera, image data representing a captured image in which an operation target device and a surrounding area of ​​the operation target device are reflected; a difference detection unit that performs a process of detecting a difference between a captured image indicated by the image data acquired by the captured image acquisition unit and a registered image that has been recorded; an image recording unit, which records the captured image as a registered image if the difference detection unit detects a difference, and obtains a second captured image that is different from the recorded registered image and records the second captured image as a registered image if the difference detection unit does not detect a difference; as well as A downloading unit is configured to download an application for operating the operation target device represented in the registration image recorded by the image recording unit.

2. The equipment operating device according to claim 1, characterized in that: The image recording unit acquires, from the camera via the captured image acquisition unit, image data indicating a captured image in which the operation target device with the identification mark and the surrounding area are projected, as the second captured image.

3. The equipment operating device according to claim 2, characterized in that: The device operating device includes a guidance prompting unit, which prompts a guide to affix the identification mark if the difference detecting unit does not detect the difference. The captured image acquisition unit acquires image data representing a captured image of the operation target device to which the identification mark is attached and the surrounding area from the camera after the guidance is prompted by the guidance prompting unit. The image recording unit acquires, as image data representing the second captured image, image data acquired by the captured image acquiring unit after the guidance is presented by the guidance presenting unit.

4. The equipment operating device according to claim 1, characterized in that: The image recording unit synthesizes an illustration component representing the identification mark for the operation object device reflected in the captured image shown by the image data obtained by the captured image obtaining unit, and obtains an image reflecting the operation object device with the identification mark and the surrounding area as the second captured image.

5. The equipment operating device according to claim 1, characterized in that: The device operating apparatus includes an operation accepting unit that accepts selection of any one operating application from among the one or more operating applications downloaded by the downloading unit and executes the selected operating application.

6. The equipment operating device according to claim 1, characterized in that: The device operating apparatus includes a grid adding unit that adds a grid to the captured image represented by the image data acquired by the captured image acquiring unit, and outputs the captured image with the grid to the difference detecting unit and the image recording unit, respectively.

7. The equipment operating device according to claim 1, characterized in that: The equipment operating device has a feature area designation accepting unit. When the difference detection unit performs detection processing on the difference between the captured image shown by the image data obtained by the captured image acquisition unit and the recorded registered image, the feature area designation accepting unit accepts the designation of a feature area, which is an area to be paid attention to in the captured image shown by the image data obtained by the captured image acquisition unit, and outputs the captured image with the designated feature area to the difference detection unit and the image recording unit, respectively.

8. The equipment operating device according to claim 7, characterized in that: When accepting the designation of a plurality of feature areas, the feature area designation accepting unit accepts the designation of a priority order for each feature area, and outputs the captured images in which the feature areas with the priority orders are designated to the difference detecting unit and the image recording unit, respectively.

9. A method for operating a device, wherein: The captured image acquisition unit acquires image data representing a captured image in which an operation target device and a surrounding area of ​​the operation target device are reflected from a camera. The difference detection unit performs a detection process of a difference between the captured image indicated by the image data acquired by the captured image acquisition unit and the recorded registered image. If the difference detection unit detects a difference, the image recording unit records the captured image as a registered image; if the difference detection unit does not detect a difference, the image recording unit obtains a second captured image that is different from the recorded registered image and records the second captured image as a registered image. The downloading unit downloads an application for operating the operation target device represented in the registration image recorded by the image recording unit.

Images