Processing device, program, and processing method

Abstract

To protect privacy of a consultor in remote consulting.SOLUTION: A customer terminal 1 comprises: an information recognition function part 26 which executes image recognition of first image information acquired from a camera 12, a document camera 16, and a scanner 17 and acquires first information being a processing result; and an image processing function part 27 which if the first information is third information indicative of information concerning privacy associated with an information definition file 52 indicative of information related with a consulting content, outputs fourth information by processing the first information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a processing device, a program, and a processing method. [Background technology]

[0002] Generally, remote consultation systems, like video call systems, are configured to provide remote support to clients by combining video and audio communication using cameras and microphones with printers and scanners. In these cases, the client prepares some kind of document or file, and in some cases remote support is provided by explaining how to write it or what parts of it are written. In these cases, the client's (client's) document may be displayed on a camera such as a document camera or network camera, or the contents of the document may be displayed on the counselor's screen using document images captured by a scanner, and the counselor may provide verbal explanations while viewing the document. Another method is for the counselor to use a paint function or other tool to mark the scanned document image while providing explanations. Patent Document 1 discloses a consulting system in which scanned files are shared between a client's terminal and a consultant's terminal. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2005-174135 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the documents that customers bring with them may contain, for example, PIN numbers, card information, or other private information, and although it is a remote consultation, the digital information captured by a camera or scanner is displayed directly to the counselor, which creates the problem that in some cases the counselor may be able to see more information than necessary unless the document is edited before being scanned.

[0005] In view of the above circumstances, an object of the present invention is to protect the privacy of a client in remote consultation. [Means for solving the problem]

[0006] The present invention solves the above problems, A processing device, a first processing unit that executes an image recognition process on the first image information acquired by the image acquisition unit and acquires first information that is a result of the process; a second processing unit that processes the first information and outputs fourth information if the first information is third information indicating privacy-related information linked to second information indicating information related to the consultation content; The processing device is characterized by comprising:

[0007] The present invention also provides A program that causes a computer to function as a processing device, The computer a first processing unit that executes image recognition processing on the first image information acquired by the image acquisition unit and acquires first information that is a processing result; a second processing unit that processes the first information and outputs fourth information if the first information is third information indicating information related to privacy linked to second information indicating information related to the consultation content; The program is characterized by functioning as follows.

[0008] The present invention also provides a processing method in a processing apparatus, comprising: The processing device includes: a first step of performing an image recognition process on first image information acquired by an image acquisition unit and acquiring first information as a result of the process; a second step of processing the first information and outputting fourth information if the first information is third information indicating privacy-related information linked to second information indicating information related to the consultation content; The processing method is characterized by executing the following. [Effects of the Invention]

[0009] According to the present invention, the privacy of the client can be protected in remote consultation. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a functional configuration diagram of a remote consultation system according to a first embodiment. [Figure 2A] FIG. 3 is a sequence diagram showing the processing of the first embodiment. [Figure 2B] FIG. 3 is a sequence diagram showing the processing of the first embodiment. [Figure 3] Examples of the My Number card screen display: (a) is the front and (b) is the back. [Figure 4] These are examples of the screen display of a My Number card with fill-in, (a) is the front side, and (b) is the back side. [Figure 5] This is an example of a driver's license screen display. [Figure 6] An example of a screen display of a driver's license with fill applied. [Figure 7] FIG. 10 is a functional configuration diagram of a remote consultation system according to a second embodiment. [Figure 8] FIG. 10 is a sequence diagram showing a process according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Each drawing is merely a schematic illustration to allow a sufficient understanding of the present invention. Therefore, the present invention is not limited to the illustrated examples. Furthermore, in the drawings, the dimensions of components constituting the present invention may be exaggerated for clarity. In addition, in each drawing, common or similar components are designated by the same reference numerals, and redundant explanations thereof will be omitted.

[0012] [Explanation regarding the premise of the present invention] The present invention relates to a function for visually protecting private information displayed on camera images and document images in a remote consultation system.

[0013] First Embodiment [composition] Fig. 1 is a functional configuration diagram of a remote consultation system according to a first embodiment. As shown in Fig. 1, the remote consultation system 100 is a system that enables online communication between a customer (client) and a counselor. The remote consultation system 100 includes a customer terminal 1, a counselor terminal 2, and an information management server 3. The customer terminal 1 and counselor terminal 2 are connected to each other so that they can communicate with each other. The customer terminal 1 and the information management server 3 are connected to each other so that they can communicate with each other.

[0014] The customer terminal 1 includes hardware such as an input unit, an output unit, a control unit, and a memory unit. For example, if the control unit is configured with a CPU (Central Processing Unit), information processing by a computer including the control unit is realized by program execution processing by the CPU. Furthermore, the memory unit included in the computer stores various programs for realizing the functions of the computer in response to instructions from the CPU. This realizes collaboration between software and hardware. The programs can be provided by recording them on a recording medium or via a network. The customer terminal 1 may be communicatively connected to a console (not shown) used by the user. The console can display the processing results of the customer terminal 1 and input requests to the customer terminal 1. The above description of the customer terminal 1 also applies to the counselor terminal 2 and the information management server 3 (and the video conference multipoint server 4, described below).

[0015] (Customer terminal 1) The customer terminal 1 is an information processing device used by a customer. The customer terminal 1 can be, for example, a PC, a smartphone, or a tablet terminal, but is not limited to these. The customer terminal 1 is equipped with a mouse 11, a camera 12, a microphone / speaker 13, a monitor 14, a printer 15, a document camera 16, a scanner 17, and a video call system 20.

[0016] The mouse 11 is an input unit that accepts input operations from the customer. The input unit that accepts input operations from the customer is not limited to the mouse 11, and may be, for example, a keyboard or a touch panel. The camera 12 is an input unit that receives images generated by a photographing process. The microphone speaker 13 is an input unit and an output unit that performs input and output processing of audio. The monitor 14 is an output unit that displays images and the like. The printer 15 is an output unit that outputs printed matter through printing processing. The document camera 16 is an input unit that accepts an image generated by photographing a document. Instead of the document camera 16, for example, a network camera may be used. The scanner 17 is an input unit that reads an image and can capture a document image captured by the document camera 16.

[0017] The video call system 20 is a functional unit for making video calls with other information processing devices. The video call system 20 includes a video processing unit 21, an audio processing unit 22, a data processing unit 23, a codec 24, a connection processing unit 25, an information recognition function unit 26, and an image processing function unit 27. The video processing unit 21 processes video data and image data. The video processing unit 21 can process video data from the camera 12. The video processing unit 21 can also process image data from the document camera 16 and the scanner 17. The video processing unit 21 can also output the processed image data and video data to the monitor 14. The audio processing unit 22 processes audio data. The audio processing unit 22 can process audio data from the microphone speaker 13. The audio processing unit 22 can also output the processed audio data to the microphone speaker 13. The data processing unit 23 processes input operations, requests, and other data. The codec 24 encrypts and decrypts image data, video data, audio data, text data, and various other types of data. The connection processing unit 25 processes sessions with other information processing devices.

[0018] The information recognition function unit 26 recognizes information contained in the acquired image data by image recognition. The information recognition function unit 26 can be, for example, but is not limited to, OCR (Optical Character Recognition). The information recognition function unit 26 can extract predetermined character information (such as a character string or a number string) and document information (such as the type of document (type of identification card)) from the image to be recognized by image recognition. The information recognition function unit 26 can combine the image data to be recognized acquired from the document camera 16 or the scanner 17 with data of the recognition result (e.g., text data) and output it to the image processing function unit 27. The information recognition function unit 26 can also output the recognition result to the information management server 3. The image processing function unit 27 processes the acquired image data and video data. Examples of processing of the image data and video data include, but are not limited to, mosaic and masking (filling in).

[0019] (Counselor terminal 2) The counselor terminal 2 is an information processing device used by the counselor. The counselor terminal 2 can be, for example, a PC, a smartphone, or a tablet terminal, but is not limited to these. The counselor terminal 2 is equipped with a mouse 31, a camera 32, a microphone / speaker 33, a monitor 34, a printer 35, and a video call system 40.

[0020] The mouse 31 is an input unit that accepts input operations from the counselor and has the same functions as the mouse 11. The camera 32, microphone speaker 33, monitor 34, and printer 35 have the same functions as the camera 12, microphone speaker 13, monitor 14, and printer 15, respectively.

[0021] The video call system 40 is a functional unit that performs video calls with other information processing devices. The video call system 40 includes a video processing unit 41, an audio processing unit 42, a data processing unit 43, a codec 44, and a connection processing unit 45. The video processing unit 41, the audio processing unit 42, the data processing unit 43, the codec 44, and the connection processing unit 45 have the same functions as the video processing unit 21, the audio processing unit 22, the data processing unit 23, the codec 24, and the connection processing unit 25, respectively.

[0022] (Information Management Server 3) The information management server 3 is an information processing device that manages various pieces of information necessary for video calls. The information management server 3 stores a database 51 and an information definition file 52. The database 51 stores various information necessary for video calls. Information stored in the database 51 includes, but is not limited to, user information and character information to be recognized. The user information is information that identifies a customer using the customer terminal 1 or a counselor using the counselor terminal 2. The user information can be represented by, for example, a combination of letters and numbers. The character information to be recognized indicates information recognized by the information recognition function unit 26. The information definition file 52 defines information that will not be disclosed to counselors in image data or video data provided by a customer. The information definition file 52 can determine the character strings and ranges to be processed by the image processing function unit 27. The information management server 3 can register definitions in the information definition file 52 for each document or piece of paper provided by a customer. The information management server 3 can also modify the definitions in the information definition file 52 to determine the information that counselors can access, depending on the type of consulting business the counselors provide and the content of the consultation from the customer. The information that can be accessed can be uniquely determined based on the type of business and the content of the consultation. Information that the customer does not want to disclose (if disclosure is not required), information that counselors cannot request disclosure of, and information that counselors do not intend to request disclosure of can also be included in the type of business and the content of the consultation, and the customer or counselor may select this information as appropriate. Note that determining the information that counselors can access and determining the information that counselors cannot access are synonymous.

[0023] [process] The processing of the first embodiment will be described. FIGS. 2A and 2B are sequence diagrams showing the processing of the first embodiment. The information definition file 52 defines information that will not be disclosed to the counselor for each document used in remote consultation. First, for example, the data processing unit 23 determines the business type and consultation content in accordance with input operations from the customer (step S1). The business type and consultation content are determined appropriately at the customer's discretion in light of the consultation that the customer brings to the counselor. The determined business type and consultation content are transmitted to the information management server 3. Next, the information management server 3 changes the definition in the information definition file 52 according to the received business type and consultation content, and determines the information that the counselor can refer to (step S3).

[0024] Next, the connection processing unit 25 of the customer terminal 1 sends a connection request to the connection processing unit 45 of the clerk terminal 2 (step A1). Next, the connection processing unit 45 sends a connection response to the connection processing unit 25 (step A2). This establishes a session between the customer terminal 1 and the clerk terminal 2. A call between the customer and the clerk begins via the video processing unit 21 and audio processing unit 22 of the customer terminal 1 and the video processing unit 41 and audio processing unit 42 of the clerk terminal 2 (step A3). The call may or may not include video.

[0025] Alternatively, instead of step S1, the data processing unit 43 may determine the business type and consultation content in accordance with input operations from the counselor before the session is established (steps A1 to A3) (step S2). In other words, the counselor can determine the information that can be referenced at their discretion before the consultation begins. Alternatively, instead of step S1, the data processing unit 43 may determine the business type and consultation content in accordance with input operations from the customer after the session is established (steps A1 to A3) (step S4). In other words, the counselor can determine the information that can be referenced at their discretion during the consultation (step S6). Alternatively, instead of step S1, the data processing unit 43 may determine the business type and consultation content in accordance with input operations from the counselor after the session is established (steps A1 to A3) (step S5). In other words, the counselor can determine the information that can be referenced at their discretion during the consultation (step S6).

[0026] Moving on to FIG. 2B, after establishing a session (steps A1 to A3), the document camera 16 captures the documents prepared by the customer and transmits the resulting image to the information recognition function unit 26 (step B1). Alternatively, the scanner 17 captures the documents prepared by the customer and transmits the captured document (image) to the information recognition function unit 26 (step B2). Next, the information recognition function unit 26 performs image recognition on the received video or image (step B3). Note that image recognition on video can be image recognition on a still image taken at a predetermined time. Image recognition can extract text information or text information from the target image. Next, the information recognition function unit 26 determines whether the extracted text information or text information contains specific text information or text information (step B4). The information recognition function unit 26 can obtain the specific text information or text information by accessing the information management server 3 and reading the information definition file 52. If specific character information or document information is included (Yes in step B5), the image processing function unit 27 performs a fill-in process on the image to be recognized (step B6). Specifically, the image processing function unit 27 applies mosaic or masking to information in the image to be recognized that cannot be referenced by the counselor, in accordance with the definition in the information definition file 52. Next, the image processing function unit 27 transmits the filled-in image to the counselor terminal 2 as a processed image (step B7). The counselor can respond to the customer's consultation by referring to information other than the unreferenced information. On the other hand, if specific character information or document information is not included (No in step B5), the information recognition function unit 26 transmits the image to be recognized to the counselor terminal 2 as an unprocessed image that has not been processed by the image processing function unit 27 (step B8).

[0027] When the document presented by the customer to the counselor is a specified document with a prescribed format, the location of information that the counselor does not want to see is also determined. Examples of specified documents include, but are not limited to, a My Number card or a driver's license. The information recognition function unit 26 can identify the type of specified document through image recognition. The information management server 3 can define in advance (even before the consultation) in the information definition file 52 where various information is to be written on the specified document, and can also define in advance (even before the consultation) information that will not be disclosed to the counselor. Therefore, even without determining the information that can be referenced (steps S3 and S6), once the information recognition function unit 26 performs image recognition of the specified document, the image processing function unit 27 can perform the fill-in process (step B6) according to the information definition file 52.

[0028] Figure 3 is an example of a screen display of a My Number card, (a) is the front side and (b) is the back side. Figure 4 is an example of a screen display of a My Number card with fill-in, (a) is the front side and (b) is the back side. Once the information recognition function unit 26 performs image recognition of the My Number card, the image processing function unit 27 can apply fill-in M1 and M2 according to the information definition file 52. Fig. 5 is an example of a screen display of a driver's license. Fig. 6 is an example of a screen display of a driver's license with fill-in. Once the information recognition function unit 26 performs image recognition of the driver's license, the image processing function unit 27 can apply fill-in M3 in accordance with the information definition file 52.

[0029] [effect] According to the first embodiment, the privacy of the client can be protected during remote consultations. More specifically, by specifying in advance information that does not need to be displayed to the counselor, and having the system automatically detect and process this information, the client's private information is not displayed more than necessary, and documents containing client information can be used without being seen by the counselor, who is a third party.

[0030] Second Embodiment In explaining the second embodiment, differences from the first embodiment will be mainly explained, and explanation of overlapping points will be omitted. The second embodiment is a remote consultation system with a multipoint configuration in which one customer is connected to multiple counselors (1:N), rather than the one-to-one connection between terminals shown in the first embodiment.

[0031] [composition] Fig. 7 is a functional configuration diagram of a remote consultation system according to the second embodiment. As shown in Fig. 7, the remote consultation system 200 is a system that enables online communication between a customer (client) and multiple counselors. The remote consultation system 200 includes a customer terminal 1, multiple counselor terminals 2, and a multipoint video conference server 4. The customer terminal 1 and the multipoint video conference server 4 are connected to each other so that they can communicate with each other. The counselor terminal 2 and the multipoint video conference server 4 are connected to each other so that they can communicate with each other.

[0032] The customer terminal 1 includes the mouse 11, camera 12, microphone / speaker 13, monitor 14, printer 15, document camera 16, scanner 17, and video call system 20, which have already been described. The video call system 20 includes the video processing unit 21, audio processing unit 22, data processing unit 23, codec 24, and connection processing unit 25, which have already been described. The counselor terminal 2 includes the mouse 31, camera 32, microphone / speaker 33, monitor 34, printer 35, and video call system 40, which have already been described. The video call system 40 includes the video processing unit 41, audio processing unit 42, data processing unit 43, codec 44, and connection processing unit 45, which have already been described.

[0033] (Videoconferencing multipoint server 4) The multipoint video conference server 4 is an information processing device that manages various information necessary for video calls and monitors the video calls between the customer terminal 1 and the counselor terminal 2. The multipoint video conference server 4 comprises a conference management function unit 61, a video / audio management unit 62, a codec 63, a data processing unit 64, an image processing unit 65, and an information management unit 66.

[0034] The conference management function unit 61 monitors the video call between the customer terminal 1 and the counselor terminal 2 and manages information required for the video call. The video / audio management unit 62 manages the video data, image data, and audio data exchanged during the video call between the customer terminal 1 and the counselor terminal 2. The codec 63 has the same functions as the codec 24 . The data processing unit 64 has the same functions as the data processing unit 23 .

[0035] The image processing unit 65 processes image data or video data from the customer terminal 1. The image processing unit 65 has the information recognition function unit 26 and the image processing function unit 27, which have already been described. The information management unit 66 manages various pieces of information necessary for video calls. The information management unit 66 stores the database 51 and the information definition file 52, which have already been described.

[0036] [process] The processing of the second embodiment will now be described. Fig. 8 is a sequence diagram showing the processing of the second embodiment. The information definition file 52 defines, for each document and for each piece of paper used in remote consultation, the information that will not be disclosed to the counselor. The processing (steps S1 to S6) for determining the information that the counselor can refer to, as explained in Fig. 2A, applies to the information definition file 52 of the multipoint video conference server 4. In addition, the session establishment (steps A1 to A3) explained in Fig. 2A is performed.

[0037] After establishing a session (steps A1 to A3), the document camera 16 captures the document prepared by the customer and sends the generated video to the information recognition function unit 26 (step C1). Alternatively, the scanner 17 captures the document prepared by the customer and sends the captured document (image) to the information recognition function unit 26 (step C2). Next, the information recognition function unit 26 performs image recognition on the received video or image (step C3). Note that image recognition on video can be image recognition on a still image taken at a predetermined time. Image recognition can extract text information or text information from the image to be recognized. Next, the information recognition function unit 26 determines whether the extracted text information or text information contains specific text information or text information (step C4). The information recognition function unit 26 can obtain the specific text information or text information by accessing the information management unit 66 and reading the information definition file 52. If specific character information or document information is included (Yes in step C5), the image processing function unit 27 performs a fill-in process on the image to be recognized (step C6). Specifically, the image processing function unit 27 applies mosaic or masking to information in the image to be recognized that cannot be referenced by counselors, in accordance with the definition in the information definition file 52. Next, the image processing function unit 27 transmits the filled-in image as a processed image to each of the multiple counselor terminals 2 (step C7). Each counselor can respond to the customer's consultation by referring to information other than the inaccessible information. On the other hand, if specific character information or document information is not included (No in step C5), the information recognition function unit 26 transmits the image to be recognized as an unprocessed image that has not been processed by the image processing function unit 27 to each of the multiple counselor terminals 2 (step C8). The same processing as in the first embodiment can be applied even when the document that the customer presents to the counselor is a specific document with a prescribed format.

[0038] [effect] The second embodiment provides the same effects as the first embodiment. In the second embodiment, when a multipoint connection is established, text detection is performed within a document on the client terminal 1 side on the multipoint video conferencing server 4, and the image is processed and distributed to each counselor terminal 2. Therefore, even when multiple counselor terminals 2 are connected, processing is not performed individually for each counselor terminal 2, and processing can proceed within the same framework as the first embodiment.

[0039] [Relationship with the claims] The customer terminal 1 in the first embodiment and the multipoint video conference server 4 in the second embodiment each correspond to a "processing device" in the claims. The camera 12, the document camera 16, and the scanner 17 each correspond to an "image reading unit" in the claims. The image captured by the camera 12, the image generated by the document camera 16, and the image captured by the scanner 17 each correspond to the "first image information" in the claims. The recognition result of the information recognition function unit 26 corresponds to the "first information" and "third information" in the claims. The information recognition function unit 26 corresponds to the "first processing unit" in the claims. The information definition file 52 corresponds to the "second information" in the claims. The processed image (steps B7 and C7) corresponds to the "fourth information" in the claims. The image processing function unit 27 corresponds to the "second processing unit" in the claims. The customer terminal 1 in the first and second embodiments corresponds to the "first terminal" in the claims. The counselor terminal 2 in the first and second embodiments corresponds to the "second terminal" in the claims.

[0040] [Variations] (a): In the first and second embodiments, a network camera or a document camera was used as an external device for capturing document images, but other external input devices can also be used as long as they can capture images into the video call system. (b): The determination of the information that the counselor can refer to (steps S3 and S6) may be realized by machine learning. For example, a learning machine can be prepared that is trained with training data in which the customer's profile and provided materials are input and the information that the counselor can refer to is output. (c): The information that the counselor can refer to may be determined by executing a process that indicates an agreement between the customer and the counselor, rather than by following the input operations from the customer (steps S1, S4) or the input operations from the counselor (steps S2, S5). (d) In the second embodiment, the information that each counselor can refer to may be different for each of the counselors at the multiple counselor terminals 2.

[0041] (e): The above-mentioned invention-specific features can be combined as appropriate. (f): Means that can be realized in software can be realized in hardware, and means that can be realized in hardware can be realized in software. [Explanation of symbols]

[0042] 100,200 Remote consultation system 1 Customer terminal 2 Counselor terminal 3. Information Management Server 4 Video Conferencing Multipoint Server 26 Information recognition function section 27 Image processing function section 52 Information definition file

Claims

1. A processing device comprising: a first processing unit that executes an image recognition process on the first image information acquired by the image acquisition unit and acquires first information that is a result of the process; a second processing unit that processes the first information and outputs fourth information if the first information is third information indicating information related to privacy linked to second information indicating information related to a consultation content; A processing device comprising:

2. There are a plurality of second terminals to which the fourth information is output; The processing device according to claim 1 , wherein the second processing unit outputs the fourth information to each of the plurality of second terminals.

3. When a connection is established between a first terminal including the image acquisition unit and a second terminal to which the fourth information is to be output, 3. The processing device according to claim 1 or claim 2, wherein the third information is information determined based on at least any of an input operation performed from the first terminal before the connection is established, an input operation performed from the first terminal after the connection is established, an input operation performed from the second terminal after the connection is established, and an input operation performed from the second terminal after the connection is established.

4. A program for causing a computer to function as a processing device, The computer, a first processing unit that executes an image recognition process on the first image information acquired by the image acquisition unit and acquires first information that is a result of the process; a second processing unit that processes the first information and outputs fourth information if the first information is third information indicating information related to privacy linked to second information indicating information related to the consultation content; A program characterized by causing the program to function as a

5. A processing method in a processing device, comprising: The processing device includes: a first step of executing an image recognition process on first image information acquired by an image acquisition unit to acquire first information that is a result of the process; a second step of processing the first information to output fourth information if the first information is third information indicating information related to privacy linked to second information indicating information related to the consultation content; A processing method comprising the steps of:

Images