Information terminal, character recognition method, and character recognition program

Abstract

And the workload of a worker when acquiring the equipment data of the field equipment through shooting is reduced. An information terminal is provided with a control unit that repeatedly performs a process for acquiring a captured frame image and a process for storing a character recognition result obtained by performing character recognition on the frame image in a storage unit, and determines the character recognition result when a predetermined operation on the information terminal is accepted.

Description

Technical Field

[0001] The invention relates to an information terminal, a character recognition method and a character recognition program. Background Art

[0002] A technique is known in which the device data of a field device is acquired by photographing a device nameplate attached to various field devices such as an air conditioner using an information terminal, etc., and performing a text recognition process. According to this technique, for example, a maintenance worker can manage the device data of a field device that has been field-surveyed simply by photographing the device nameplate.

[0003] Patent Document 1: Japanese Patent Application Publication No. 2020-091748 Summary of the invention

[0004] <Problems to be Solved by the Invention>

[0005] On the other hand, when shooting on site, depending on the shooting conditions, the text recognition process may be performed inappropriately, resulting in erroneous recognition, which may result in failure to obtain correct device data for part or all of the device data. In this case, the maintenance staff needs to reshoot, which increases the workload for the maintenance staff.

[0006] The present invention can reduce the workload of workers when acquiring equipment data of on-site equipment by photographing.

[0007] <Methods used to solve the problem>

[0008] A first aspect of the present invention is an information terminal having a control unit, wherein:

[0009] The control unit:

[0010] Repeating the process of acquiring the captured frame image and storing the character recognition result obtained by performing character recognition on the frame image in the storage unit,

[0011] When a predetermined operation on the information terminal is accepted, a character recognition result is determined.

[0012] According to the first aspect of the present invention, it is possible to reduce the workload of a worker when acquiring device data of an on-site device by imaging.

[0013] Furthermore, a second aspect of the present invention is the information terminal according to the first aspect, wherein:

[0014] When the captured frame image is displayed in real time, a character recognition result obtained by performing the character recognition is displayed in an overlapping manner.

[0015] Furthermore, a third aspect of the present invention is the information terminal according to the second aspect, wherein:

[0016] The control unit:

[0017] At a first time interval, the frame image to be displayed is updated,

[0018] The character recognition result displayed in an overlapping manner is updated at a second time interval that is longer than the first time interval.

[0019] Furthermore, a fourth aspect of the present invention is the information terminal according to any one of the first to third aspects, wherein:

[0020] The control unit:

[0021] When the captured frame image includes a plurality of items to be stored as text recognition results, the text recognition results of the items obtained by performing the text recognition are stored in the storage unit in units of items.

[0022] When a predetermined operation on the information terminal is accepted, the character recognition result is determined in units of items.

[0023] Furthermore, a fifth aspect of the present invention is the information terminal according to any one of the first to fourth aspects, wherein:

[0024] The control unit:

[0025] When the displayed character recognition result is corrected in accordance with the determined character recognition result being displayed on the display unit, the corrected character recognition result is stored in the storage unit.

[0026] Furthermore, a sixth aspect of the present invention is the information terminal according to any one of the first to fifth aspects, wherein:

[0027] The control unit:

[0028] By performing text recognition on a frame image of a nameplate of a device, a model name and a production number are obtained as text recognition results.

[0029] Furthermore, a seventh aspect of the present invention is the information terminal according to any one of the first to sixth aspects, wherein:

[0030] The control unit:

[0031] storing the character recognition result obtained by performing the character recognition in the first storage unit,

[0032] The character recognition result determined by accepting a predetermined operation on the information terminal is stored in the second storage unit.

[0033] Furthermore, an eighth aspect of the present invention is the information terminal according to the seventh aspect, wherein:

[0034] The control unit:

[0035] Whenever a character recognition result is obtained by performing the character recognition, the character recognition result stored in the first storage unit is updated.

[0036] Furthermore, a ninth aspect of the present invention is the information terminal according to any one of the first to eighth aspects, wherein:

[0037] The character recognition result is a result determined by the control unit to satisfy a storage condition among the results obtained by performing the character recognition.

[0038] Furthermore, a tenth aspect of the present invention is the information terminal according to the ninth aspect, wherein:

[0039] The control unit:

[0040] When a result obtained for a predetermined item by performing the character recognition is a preset character string, it is determined that the storage condition is satisfied, and the result obtained for the predetermined item is stored in the storage unit as a character recognition result for the predetermined item.

[0041] Furthermore, an eleventh aspect of the present invention is the information terminal according to the seventh aspect, wherein:

[0042] The control unit:

[0043] The frame image for which the character recognition result stored in the first storage unit is obtained is stored in the first storage unit in association with the character recognition result.

[0044] The frame image for which the character recognition result stored in the second storage unit is obtained is stored in the second storage unit in association with the character recognition result.

[0045] In addition, a twelfth aspect of the present invention is a character recognition method, wherein the control unit of the information terminal executes the following steps:

[0046] Repeating the process of acquiring the captured frame image and storing the character recognition result obtained by performing character recognition on the frame image in the storage unit; and

[0047] When a predetermined operation on the information terminal is accepted, a character recognition result is determined.

[0048] According to the twelfth aspect of the present invention, it is possible to reduce the workload of a worker when acquiring device data of an on-site device by photographing it.

[0049] Furthermore, a thirteenth aspect of the present invention is a program for causing a control unit of an information terminal to execute the following steps:

[0050] Repeating the process of acquiring the captured frame image and storing the character recognition result obtained by performing character recognition on the frame image in the storage unit; and

[0051] When a predetermined operation on the information terminal is accepted, a character recognition result is determined.

[0052] According to the thirteenth aspect of the present invention, it is possible to reduce the workload of a worker when acquiring device data of an on-site device by photographing it. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a diagram showing an example of use of an information terminal.

[0054] Figure 2 This is a diagram showing an example of a character recognition result obtained by performing a character recognition process on each frame image.

[0055] Figure 3 This is a diagram showing an example of the hardware configuration of an information terminal.

[0056] Figure 4 FIG. 1 is a first diagram showing an example of a functional structure of an information terminal.

[0057] Figure 5 This is a diagram showing an example of the overall operation of the information terminal.

[0058] Figure 6 It is a diagram showing a specific example of the processing from the recognition unit to the extraction unit.

[0059] Figure 7 It is a diagram showing a specific example of the processing of the determination unit.

[0060] Figure 8 This is a flowchart showing the processing flow of the first storage control unit.

[0061] Fig. 9 This is a flowchart showing the processing flow of the second storage control unit.

[0062] Fig. 10A FIG. 1 is a first diagram showing a display example of a frame image and a frame displayed on the display unit.

[0063] Fig. 10BFIG. 2 is a second diagram showing a display example of a frame image and a frame displayed on the display unit.

[0064] Fig.11 This is a flowchart showing the flow of character recognition processing executed by an information terminal.

[0065] Fig.12 FIG. 122 is a diagram showing an example of a functional structure of an information terminal. DETAILED DESCRIPTION

[0066] Hereinafter, each embodiment will be described with reference to the drawings. It should be noted that in this specification and the drawings, components having substantially the same functional structure are given the same reference numerals, and repeated descriptions are omitted.

[0067] [First embodiment]

[0068] <Examples of using information terminals>

[0069] First, a usage example of the information terminal according to the first embodiment will be described. Figure 1 FIG. 1 is a diagram showing an example of using an information terminal. Figure 1 As shown on the left side of FIG. 1 , the information terminal 120 is used, for example, in a scenario where device data is read from a device nameplate 111 attached to a side surface of a field device 110 or the like.

[0070] According to the information terminal 120, for example, when the maintenance worker 130 conducts a field survey and registers the survey results in a server device (not shown) or the like, the maintenance worker 130 can save the trouble of manually inputting the device data for specifying the field device 110 to be surveyed. In addition, the maintenance worker 130 can be prevented from erroneously inputting the device data.

[0071] It should be noted that in this embodiment, it is assumed that the field device 110 is an outdoor unit of an air conditioner and the information terminal 120 is a portable terminal. However, the field device 110 is not limited to the outdoor unit of the air conditioner and may be a unit other than the outdoor unit of the air conditioner, or may be a field device other than the air conditioner. In addition, the information terminal 120 is not limited to a portable terminal and may be an information terminal other than a portable terminal, as long as it has an optical character recognition function.

[0072] Figure 1 The right side of 1 shows that the maintenance worker 130 photographed the equipment nameplate 111 and the photographed image 121 is displayed on the display screen of the information terminal 120. Figure 1 As shown on the right side of , the captured image 121 includes an image area 122 corresponding to the device nameplate 111 .

[0073] When the field device 110 is an outdoor unit of an air conditioning device, the device nameplate 111 displayed in the image area 122 includes, for example:

[0074] ·Indicates that it is a record of an outdoor unit;

[0075] ·Indicates that it is an outdoor unit that can be used for both cooling and heating;

[0076] ·Records about the model (label, model code data);

[0077] Records of production numbers (labels, production number data);

[0078] Records on product quality (labels, quality data);

[0079] ·Records of power supply specifications (nameplate, phase data, voltage data, frequency data);

[0080] · Records on the output of the compressor motor (nameplate, power consumption data);

[0081] · Records of management numbers (labels, management code data);

[0082] ·Records on the quantity and type of refrigerant (label, refrigerant quantity data, refrigerant type code);

[0083] ·Records of design pressure (label, pressure data);

[0084] Records on the production date (label, production date data);

[0085] etc. Figure 1 In the figure, omission is indicated by "*". ). In addition, the actual description of the device nameplate 111 includes many types of characters, such as letters, numbers, symbols, and Chinese characters (the language of the country where the field device 110 is installed).

[0086] It should be noted that in this embodiment, when the field device 110 is photographed by the information terminal 120, the period from when the maintenance worker 130 starts photographing to when the shutter is operated is photographed as dynamic image data. In addition, in this embodiment, text recognition processing is performed on each frame image included in the dynamic image data.

[0087] <An Example of Character Recognition Processing>

[0088] Next, a description will be given of a character recognition result when the maintenance worker 130 captures the field device 110 as moving image data and the information terminal 120 performs character recognition processing on each frame of the image. Figure 2This is a diagram showing an example of a character recognition result obtained by performing a character recognition process on each frame image.

[0089] exist Figure 2 , the vertical axis 201 represents the time axis, Figure 2 The example shows frame images (reference numerals 211 to 213 ) captured at times T1 , T2 , and T3 among the frame images included in the moving image data.

[0090] also, Figure 2 The example shows that by performing character recognition processing on each frame image included in the dynamic image data, the character recognition results of the model name, production number, and production date of the field device 110 are obtained as device data (symbols 221 to 223).

[0091] Specifically, Figure 2 The example shows that in the case of frame image 1 captured at time = T1, the character recognition result is obtained by performing character recognition processing on the model name and production number. On the other hand, Figure 2 The example shows that no text recognition result is obtained by performing text recognition processing on the production date.

[0092] also, Figure 2 The example shows that in the case of frame image 2 captured at time = T2, a character recognition result is obtained by performing character recognition processing on any one of the model name, production number, and production date, and the device data is acquired.

[0093] also, Figure 2 The example shows that in the case of frame image 3 captured at time = T3, the text recognition result is obtained by performing text recognition processing on the production number. On the other hand, Figure 2 The example shows that no text recognition result can be obtained by performing text recognition processing on the model name and production date.

[0094] As described above, when the field device 110 is photographed as moving image data, due to slight differences in the photographing conditions during the photographing period that the maintenance worker 130 does not notice, there are cases where a character recognition result can be obtained or a case where a character recognition result cannot be obtained.

[0095] In other words, at the time of the shutter operation, the possibility of obtaining text recognition results for all multiple object items (multiple items to be recognized) is not high. Therefore, in the information terminal 120 of this embodiment, the text recognition process is performed on each frame image captured before the shutter operation. Then, in the information terminal 120 of this embodiment, when text recognition results are obtained for all object items in any frame image, the text recognition result is obtained as device data.

[0096] This can improve the possibility of acquiring device data compared to the case where only the text recognition process is performed on the frame image acquired at the time of the shutter operation. Therefore, according to the information terminal 120 of this embodiment, the workload of maintenance personnel when acquiring device data of field devices by photographing can be reduced.

[0097] <Hardware structure of information terminal>

[0098] Next, the hardware configuration of the information terminal 120 will be described. Figure 3 FIG. 1 is a diagram showing an example of the hardware structure of an information terminal. Figure 2 As shown, the information terminal 120 includes a processor 301, a memory 302, an auxiliary storage device 303, a user interface device 304, a camera device 305 and a communication device 306. It should be noted that the hardware of the information terminal 120 is connected to each other via a bus 307.

[0099] The processor 301 includes various computing devices such as a CPU (Central Processing Unit) and reads various programs (eg, a text recognition program described later) to the memory 302 and executes them.

[0100] The memory 302 includes a main storage device such as a ROM (Read Only Memory) and a RAM (Random Access Memory). The processor 301 and the memory 302 form a so-called computer, and the computer realizes various functions by executing various programs read out to the memory 302 by the processor 301.

[0101] The auxiliary storage device 303 stores various programs and various data to be used when the processor 301 executes the various programs (for example, frame images, libraries, correction tables, device data rules, character recognition results, etc. described later).

[0102] The user interface device 304 has, for example, an operation function for executing various operations of the maintenance worker 130 and a display function for displaying processing results of various processes executed by the information terminal 120 .

[0103] The imaging device 305 captures the device nameplate 111 and generates the captured image 121. The communication device 306 is a communication device for communicating with an external device (for example, a server device not shown) via a network.

[0104] It should be noted that various programs installed in the auxiliary storage device 303 are installed by downloading from the network via the communication device 306, for example.

[0105] <Functional structure of information terminal>

[0106] Next, the functional configuration of the information terminal 120 will be described. Figure 4 FIG1 is a first diagram showing an example of the functional structure of an information terminal. As described above, a character recognition program is installed in the information terminal 120, and by executing the program, the information terminal 120 performs the following functions:

[0107] Operation receiving unit 401,

[0108] Acquisition unit 402,

[0109] Identification unit 403,

[0110] ·Revision section 404,

[0111] Extraction unit 405,

[0112] ·Determination unit 406,

[0113] ·Bounding box generation unit 407,

[0114] · First storage control unit 408,

[0115] · Second storage control unit 409,

[0116] Display unit 410,

[0117] Manual correction unit 411.

[0118] The operation accepting unit 401 accepts a photographing start operation and a shutter operation among various operations performed by the maintenance worker 130 on the information terminal 120. When the photographing start operation is accepted, the operation accepting unit 401 notifies the acquiring unit 402 of a photographing start instruction.

[0119] On the other hand, when a shutter operation is accepted, the operation accepting unit 401 notifies the acquiring unit 402 of an instruction to end photographing, and notifies the second storage control unit 409 that a shutter operation has been performed.

[0120] When the operation receiving unit 401 notifies the start instruction of shooting, the acquisition unit 402 activates the camera device 305 and starts to acquire the captured image data captured as a dynamic image at a predetermined frame period (an example of a predetermined period). The acquisition unit 402 notifies the display unit 410 of each frame image contained in the acquired captured image data, and extracts each frame image at intervals and notifies the recognition unit 403. Furthermore, the acquisition unit 402 stores each frame image notified to the recognition unit 403 in the frame image storage unit 421. It should be noted that these processes of the acquisition unit 402 are repeatedly performed until the shooting end instruction is notified from the operation receiving unit 401.

[0121] The recognition unit 403 performs character recognition processing on each frame image notified by the acquisition unit 402 while referring to the library storage unit 422, and notifies the recognized characters to the correction unit 404. The library storage unit 422 stores a library for character recognition of letters, numbers, and symbols.

[0122] The correction unit 404 identifies the characters that are incorrectly recognized in the character recognition process among the characters notified by the recognition unit 403 while referring to the correction table storage unit 423, and corrects the recognized characters to correct characters. In addition, the correction unit 404 notifies the corrected characters to the extraction unit 405. It should be noted that the correction table storage unit 423 stores a correction table in which characters that are easily incorrectly recognized when performing character recognition processing on the frame image included in the captured image data obtained by capturing the nameplate of the device are stored in association with the characters when the characters are correctly recognized.

[0123] The extraction unit 405 extracts a character string from the corrected characters notified by the correction unit 404, and notifies the extracted character string together with their respective positions and sizes to the determination unit 406. It should be noted that, for the corrected characters, the extraction unit 405 extracts one line as one character string. When a blank space of a predetermined number of characters is included, one line is determined by line break. However, in this embodiment, when there is a blank space after a number, the extraction unit 405 does not break the line.

[0124] The determination unit 406 determines whether the text string notified from the extraction unit 405 is described (whether the storage condition is satisfied) in accordance with the naming rules of the target items (in this embodiment, the items whose identification objects are the model name, production number, and production date). It should be noted that the naming rules that define the character type, number of characters, and sequence of the text string constituting the target item are stored in the device data rule storage unit 424, and the determination unit 406 refers to the device data rule storage unit 424 to determine whether the description is performed in accordance with the naming rules of the target item.

[0125] If the result of the determination by the determination unit 406 is that the description is made according to the naming rule of the object item,

[0126] · a character string (character recognition result) determined to be described according to the naming rule from among the character strings notified by the extraction unit 405; and

[0127] The position and size of the text string,

[0128] The frame generation unit 407 is notified.

[0129] Furthermore, if the result of the determination by the determination unit 406 is that all the object items are described according to the corresponding naming rule, each character string (each character recognition result) described according to the corresponding naming rule is associated with each object item and notified to the first storage control unit 408. Furthermore, the determination unit 406 notifies the first storage control unit 408 of the position and size notified by the extraction unit 405 in association with each object item.

[0130] The frame generation unit 407 generates a frame (information indicating the position and size of the character string) based on the position and size of the character string notified by the determination unit 406. In addition, the frame generation unit 407 notifies the display unit 410 of the generated frame together with the character string notified by the determination unit 406 (character recognition result).

[0131] When the character recognition results for all the target items are notified by the determination unit 406, the first storage control unit 408 stores the positions and sizes in association with the target items as device data in the first storage unit 425. When the character recognition results for all the target items are notified by the determination unit 406, the first storage control unit 408 reads the corresponding frame images from the frame image storage unit 421 and stores them in the first storage unit 425.

[0132] It should be noted that, when the text recognition result for the object item has already been stored in the first storage unit 425 together with the corresponding position and size and the frame image, the first storage control unit 408 overwrites and stores the text recognition result. Thus, the latest text recognition result among the text recognition results for all the object items repeatedly notified during the shooting period is stored in the first storage unit 425 together with the corresponding position and size and the frame image.

[0133] When the operation receiving unit 401 notifies that a shutter operation has been performed, the second storage control unit 409 reads out the latest character recognition result, the corresponding position and size, and the frame image from the first storage unit 425, and stores them in the second storage unit 426. Thus, the second storage control unit 409 can store the determined character recognition result, etc. in the second storage unit 426. In other words, in this embodiment, the shutter operation can be regarded as an operation for determining the character recognition result, etc.

[0134] Furthermore, the second storage control unit 409 notifies the frame generation unit 407 of the determined character recognition result, position, and size. Furthermore, the second storage control unit 409 notifies the display unit 410 of the determined frame image.

[0135] Furthermore, when a corrected character string obtained by manually correcting the character recognition result by the manual correction unit 411 in accordance with the notification of the determined character recognition result, position, size, and frame image is notified, the second storage control unit 409 receives the corrected character string. Furthermore, the second storage control unit 409 overwrites and stores the received corrected character string in the second storage unit 426.

[0136] The display unit 410 displays each frame image notified by the acquisition unit 402 at a predetermined frame cycle on the user interface device 304. It should be noted that when the display unit 410 displays each frame image notified at a predetermined frame cycle, the frame notified by the frame generation unit 407 and the text recognition result are overlapped. The frame and the text recognition result overlapped on each frame image by the display unit 410 are updated at a cycle longer than the predetermined frame cycle. This is because, as described above, the acquisition unit 402 extracts each frame image at intervals and notifies the recognition unit 403, so the cycle of generating the frame is longer than the predetermined frame cycle.

[0137] Since the shutter operation is performed, the display unit 410 superimposes the frame and the character recognition result notified from the frame generation unit 407 on the frame image notified from the second storage control unit 409. The display unit 410 displays the frame image superimposed with the frame and the character recognition result on the user interface device 304.

[0138] That is, the display unit 410

[0139] · Display the frame image in real time during shooting, and update the frame and text recognition results superimposed on the target item at a period longer than the specified frame period.

[0140] When the shutter operation is performed, the determined frame image, the frame generated based on the determined position and size, and the determined text recognition result are displayed.

[0141] When a correction instruction is manually inputted in response to the display unit 410 displaying the determined character recognition result and the determined frame image due to the shutter operation, the manual correction unit 411 receives the correction instruction and notifies the second storage control unit 409 of the corrected character string after the manual correction.

[0142] <Overall Operation Example of Information Terminal>

[0143] Next, an example of the overall operation of the information terminal 120 will be described. Figure 5 This is a diagram showing an example of the overall operation of the information terminal.

[0144] exist Figure 5 In FIG. 4 , “frame image 1” to “frame image 5” represent frame images acquired by the acquisition unit 402. Specifically, it represents a case where “frame image 1” is acquired by inputting a shooting start instruction, and a shutter operation is performed immediately after “frame image 5” is shot.

[0145] also, Figure 5 The example shows a situation in which “frame image 1”, “frame image 3”, and “frame image 5” are stored in the frame image storage unit 421.

[0146] It should be noted that according to Figure 5 In the example, the frame image displayed in the user interface device 304 of the information terminal 120 is updated at the first time interval a. In addition, the frame and the character recognition result displayed overlapping with the frame image in the user interface device 304 of the information terminal 120 are updated at least at the second time interval b (a<b).

[0147] In addition, Figure 5 In the example, "recognition result 1 (NG)" indicates that the recognition unit 403 to the determination unit 406 have processed "frame image 1" and determined that any of the character strings included in "frame image 1" is not described in accordance with the naming rule of the object item. In this case, "recognition result 1 (NG)" and "frame image 1" are not stored in the first storage unit 425.

[0148] On the other hand, Figure 5 In the example, "recognition result 3 (OK)" indicates that the recognition unit 403 to the determination unit 406 have processed "frame image 3" and determined that the text string contained in "frame image 3" is described in accordance with the naming rule of the object item. At this time, the first storage unit 425 stores "recognition result 3 (OK)" (text recognition result) and "frame image 3" (it should be noted that in Figure 5 In the example, position and size are omitted).

[0149] In addition, Figure 5In the example, "recognition result 5 (OK)" indicates that the recognition unit 403 to the determination unit 406 have processed "frame image 5" and determined that the character string contained in "frame image 5" is described in accordance with the naming rule of the object item. At this time, "recognition result 5 (OK)" (character recognition result) and "frame image 5" are overwritten and stored in the first storage unit 425.

[0150] also, Figure 5 The example shows a situation where “recognition result 5 (OK)” (character recognition result) and “frame image 5” stored in the first storage unit 425 are stored in the second storage unit 426 at the time when the shutter operation is performed.

[0151] <Details of processing by each functional unit of the information terminal>

[0152] Next, details of the processing of each functional unit of the information terminal (here, the identification unit 403, the correction unit 404, the extraction unit 405, the determination unit 406, the first storage control unit 408, and the second storage control unit 409) will be described.

[0153] (1) Details of the processing from the identification unit 403 to the extraction unit 405

[0154] First, the details of the processing from the identification unit 403 to the extraction unit 405 will be described. Figure 6 It is a diagram showing a specific example of the processing from the recognition unit to the extraction unit.

[0155] As described above, the library storage unit 422 stores a library for performing text recognition on letters, numbers, and symbols. Therefore, when performing text recognition processing on a frame image, when letters, numbers, and symbols are included in the frame image, the recognition unit 403 performs text recognition on the letters, numbers, and symbols in the frame image. In addition, when a frame image includes text other than letters, numbers, and symbols, the recognition unit 403 recognizes the text as any one of letters, numbers, and symbols. Specifically, when a frame image includes text other than letters, numbers, and symbols, the recognition unit 403 recognizes the text as letters, numbers, and symbols similar to the text.

[0156] exist Figure 6 In the figure, reference numeral 610 represents text obtained by the recognition unit 403 performing text recognition processing on the frame image.

[0157] Then, the correction unit 404 determines the characters that are incorrectly recognized in the characters indicated by reference numeral 610. Specifically, when the characters indicated by reference numeral 610 include letters, numbers, and symbols that are not used as target items, the correction unit 404 determines that these letters, numbers, and symbols are incorrectly recognized and corrects them.

[0158] As described above, the correction unit 404 refers to the correction table (an example of a text correction rule, for example, the correction table 600) stored in the correction table storage unit 423 to correct the letters, numbers, and symbols that are determined to be incorrectly recognized. It should be noted that, as described above, in the correction table 600, characters that are easily incorrectly recognized (characters before correction) are pre-associated with characters when the characters are correctly recognized (characters after correction). Figure 6 As shown, reference numeral 620 indicates corrected text after the incorrectly recognized text in the text shown by reference numeral 610 is corrected (it should be noted that reference numeral 620 indicates a state where a line is wrapped in order to extract one line as one text string due to paper space reasons).

[0159] Then, the extracting unit 405 sequentially extracts one line of the characters indicated by reference numeral 620 as one character string, and notifies the determining unit 406 of the extraction line. Figure 6 , reference numeral 630 indicates a case where "ARR63UP" is extracted as a text string from the text indicated by reference numeral 620.

[0160] Furthermore, the extracting unit 405 sequentially calculates the position and size of the extracted character string in the frame image before line break, and notifies the determining unit 406 of the position and size together with the extracted character string (reference numeral 630 ). Figure 6 6, reference numeral 631 indicates a case where the position and size of the extracted text string (reference numeral 630) within the frame image before line break are calculated.

[0161] (2) Details of the Processing of the Determination Unit 406

[0162] Next, details of the processing of the determination section 406 will be described. Figure 7 It is a diagram showing a specific example of the processing of the determination unit.

[0163] As described above, the naming rule 700 of the target item is stored in the device data rule storage unit 424. By referring to the naming rule 700 stored in the device data rule storage unit 424, the determination unit 406 determines whether the character string notified sequentially from the extraction unit 405 is described in accordance with the naming rule of the target item (see reference numeral 630).

[0164] Figure 7 The example of shows a case where the text string indicated by reference numeral 630 is notified, and determination unit 406 refers to the determination criterion set based on the naming rule of "model code data" in naming rule 700. Determination unit 406 determines that the text string indicated by reference numeral 630, i.e., "ARR63UP", is consistent with the determination criterion and is therefore described in accordance with the naming rule of "model code data".

[0165] Therefore, the determination unit 406 notifies the frame generation unit 407 of the character string (character recognition result) of "Model Name" and its position and size (see reference numerals 710 and 711). Figure 7 (not shown in the figure).

[0166] It should be noted that in Figure 7 In the example, it is assumed that, in the character string notified sequentially to the determination unit 406,

[0167] Although there is

[0168] A character string that matches the determination criteria set based on the naming rule of "model code data" in the naming rule 700,

[0169] But it doesn't exist

[0170] A character string that matches the judgment criteria set based on the naming rule of "production number data" in the naming rule 700, and

[0171] A character string that matches the determination criterion set based on the naming rule of “manufacturing date data” in the naming rule 700 .

[0172] Therefore, the determination unit 406 notifies the first storage control unit 408 of the recognition result (NG).

[0173] On the other hand, Figure 7 Although not shown in the example, it is assumed that in the text string notified to the determination unit 406 in sequence,

[0174] Both exist

[0175] A character string that matches the determination criteria set based on the naming rule of "model code data" in the naming rule 700,

[0176] A character string that matches the judgment criteria set based on the naming rule of "production number data" in the naming rule 700, and

[0177] A character string that matches the determination criterion set based on the naming rule of “manufacturing date data” in the naming rule 700 .

[0178] In this case, the determination unit 406 will

[0179] · Regarding the text recognition results of all target items (Recognition Result (OK)),

[0180] The position and size of the text string of "model name" (text recognition result), the text string of "production number" (text recognition result), and the text string of "production date" (text recognition result)

[0181] The first storage control unit 408 is notified.

[0182] (3) Details of Processing by the First Storage Control Unit 408

[0183] Next, details of the processing of the first storage control unit 408 will be described. Figure 8 This is a flowchart showing the processing flow of the first storage control unit.

[0184] like Figure 8 As shown, in step S801 , the first storage control unit 408 obtains the character recognition result (recognition result (OK)), the position and size, or the recognition result (NG) from the determination unit 406 .

[0185] In step S802 , the first storage control unit 408 refers to the frame image storage unit 421 and acquires a frame image corresponding to the acquired character recognition result.

[0186] In step S803, when the first storage control unit 408 acquires the character recognition result (recognition result (OK)), position, and size, it associates the acquired character recognition result with the object item and stores it as device data in the first storage unit 425. In addition, the first storage control unit 408 also associates the acquired position and size with the object item and stores them in the first storage unit 425. In addition, the first storage control unit 408 also associates the acquired frame image with the acquired character recognition result and stores it in the first storage unit 425.

[0187] It should be noted that, when the character recognition result, position, size, and frame image are already stored in the first storage unit 425, the first storage control unit 408 overwrites them for storage.

[0188] (4) Details of Processing by the Second Storage Control Unit 409

[0189] Next, the details of the processing of the second storage control unit 409 will be described. Fig. 9 This is a flowchart showing the processing flow of the second storage control unit.

[0190] like Fig. 9 As shown, in step S901, the second storage control unit 409 determines the latest character recognition result stored in the first storage unit 425 as device data.

[0191] In step S902 , the second storage control unit 409 reads out the determined character recognition result, the corresponding position and size, and the corresponding frame image from the first storage unit 425 , and stores them in the second storage unit 426 , thereby determining the character recognition result and the like.

[0192] In step S903 , the second storage control unit 409 notifies the display unit 410 of the determined frame image stored in the second storage unit 426 .

[0193] In step S904 , the second storage control unit 409 notifies the frame generation unit 407 of the determined character recognition result, position, and size stored in the second storage unit 426 .

[0194] <Display Example of Display Section>

[0195] Next, a display example of a frame image and a frame displayed on the display unit 410 will be described. Fig. 10A and Fig. 10B The first and second figures show display examples of a frame image and a frame displayed on the display unit.

[0196] Fig. 10A The example shows that when the maintenance worker 130 performs the imaging start operation, the frame image 1 shown by the reference numeral 1011 is displayed at time = T1. Fig. 10A The example shows that in the text recognition processing of frame image 1, the text recognition results are obtained for "model name" and "production number", but the text recognition result is not obtained for "production date" (refer to reference numeral 1021). Therefore, in frame image 1, a frame is displayed at the text strings of "model name" and "production number", and the text recognition result is displayed.

[0197] also, Fig. 10A The example shows a situation where the frame image 2 shown by reference numeral 1012 is displayed at time = T2. Since the text recognition process is not performed on the frame image 2, the text recognition result obtained by performing the text recognition process on the frame image 1 is displayed as it is. That is, in the frame image 2, a frame is displayed at the text string of "model name" and "production number", and the text recognition result is displayed.

[0198] also, Fig. 10A The example of 1013 shows that the frame image 3 shown at time = T3 is displayed. It means that in the text recognition process for frame image 3, the text recognition results for "model name", "production number", and "production date" are obtained (refer to reference numeral 1023). Therefore, in frame image 3, a frame is displayed at the text string of "model name", "production number", and "production date", and the text recognition result is displayed.

[0199] then, Fig. 10BThe example shows a case where the frame image 4 shown by reference numeral 1014 is displayed at time = T4. Since the text recognition process is not performed on the frame image 4, the text recognition result obtained by performing the text recognition process on the frame image 3 is displayed as it is. That is, in the frame image 4, a frame is displayed at the text string of "model name", "production number", and "production date", and the text recognition result is displayed.

[0200] also, Fig. 10B The example of 1015 shows a case where the frame image 5 shown by reference numeral 1015 is displayed at time = T5. It means that in the character recognition processing of the frame image 5, the character recognition results for "model name", "production number", and "production date" are obtained (refer to reference numeral 1025). Therefore, in the frame image 5, a frame is displayed at the character string of "model name", "production number", and "production date", and the character recognition result is displayed.

[0201] also, Fig. 10B The example of represents a situation where the maintenance worker 130 performs a shutter operation immediately after time = T5. At this time, the text recognition result obtained by performing text recognition processing on frame image 5 is stored in the first storage unit 425 as the latest text recognition result, and frame image 5 is stored as the corresponding frame image. In addition, the latest text recognition result and the frame image 5 are stored in the second storage unit 426. Therefore, after the shutter operation is performed, the frame image, frame, and text recognition result (reference numeral 1016) that are the same as the frame image, frame, and text recognition result displayed at time = T5 are displayed.

[0202] <Text recognition processing>

[0203] Next, the flow of the character recognition processing of the entire information terminal 120 will be described. Fig.11 This is a flowchart showing the flow of character recognition processing in an information terminal.

[0204] In step S1101 , the information terminal 120 receives an operation to start photographing from the maintenance worker 130 .

[0205] In step S1102 , the information terminal 120 starts capturing moving image data, acquires frame images, and displays the acquired frame images.

[0206] In step S1103 , the information terminal 120 thins out the acquired frame images and stores them in the frame image storage unit 421 .

[0207] In step S1104 , the information terminal 120 performs a text recognition process by performing a thinning operation on the acquired frame image.

[0208] In step S1105, information terminal 120 corrects the characters recognized by performing the character recognition process.

[0209] In step S1106, information terminal 120 extracts a character string from the corrected characters.

[0210] In step S1107 , the information terminal 120 determines whether the extracted character string is described in accordance with the naming rule of the target item.

[0211] In step S1108 , when information terminal 120 determines that the description is not in accordance with the naming rule (“No” in step S1108 ), the process proceeds to step S1110 .

[0212] On the other hand, in step S1108, when information terminal 120 determines that the description is in accordance with the naming rule (if "YES" in step S1108), the process proceeds to step S1109.

[0213] In step S1109 , the information terminal 120 displays the character string (character recognition result) determined to be described according to the naming rule together with a frame.

[0214] In step S1110, the information terminal 120 performs the first storage control process. It should be noted that the details of the first storage control process have been described in detail. Figure 8 The description has been given, so the description is omitted here.

[0215] In step S1111, information terminal 120 determines whether a shutter operation has been performed. If it is determined in step S1111 that a shutter operation has not been performed (in the case of "No" in step S1111), the process returns to step S1102.

[0216] On the other hand, when it is determined in step S1111 that the shutter operation has been performed (in the case of YES in step S1111 ), the process proceeds to step S1112 .

[0217] In step S1112, the information terminal 120 performs the second storage control process. It should be noted that the details of the second storage control process have been described in detail in Fig. 9 The description has been given, so the description is omitted here.

[0218] In step S1113 , the information terminal 120 displays the determined text recognition result together with the frame in an overlapping manner on the determined frame image.

[0219] <Conclusion>

[0220] As is apparent from the above description, the information terminal 120 of the first embodiment

[0221] The process of acquiring each frame image included in the captured moving image data and the process of storing the character recognition results acquired by performing the character recognition process on the frame images in the first storage unit are repeated.

[0222] When a shutter operation is performed by a maintenance worker, the character recognition result is confirmed.

[0223] Thus, in the shooting of dynamic image data, character recognition processing is performed on each frame image, and the correctly acquired character recognition result is determined at the time point of the shutter operation. Thus, according to the first embodiment, the maintenance staff does not need to re-shoot for acquiring device data.

[0224] Therefore, according to the first embodiment, it is possible to reduce the workload of a worker when acquiring device data of an on-site device by imaging.

[0225] [Second embodiment]

[0226] In the first embodiment, the maintenance worker can perform the shutter operation regardless of whether the device data can be correctly acquired during the shooting of the dynamic image data, but the shutter operation can also be disabled before the device data can be correctly acquired. The second embodiment will be described below with the differences from the first embodiment as the center.

[0227] <Functional structure of information terminal>

[0228] Fig.12 FIG. 2 is a diagram showing an example of a functional structure of an information terminal. Figure 4 The functional configuration of the information terminal 120 of the first embodiment described above is different in that the function of the operation accepting unit 1201 is different from the function of the operation accepting unit 401 .

[0229] In the information terminal 120 according to the second embodiment, the operation accepting unit 1201 accepts the imaging start operation performed by the maintenance worker 130 , notifies the acquisition unit 402 of the imaging start instruction, and starts monitoring the first storage unit 425 .

[0230] Specifically, the operation accepting unit 1201 monitors whether the character recognition result about the target item (model name, production number, production date) is stored in the first storage unit 425. When it is determined that the character recognition result is not stored, the operation accepting unit 1201 does not accept the shutter operation performed by the maintenance worker 130. On the other hand, when it is determined that the character recognition result is stored, the operation accepting unit 1201 accepts the shutter operation performed by the maintenance worker 130 and notifies the second storage control unit 409 that the shutter operation has been performed.

[0231] Summary

[0232] As is apparent from the above description, the information terminal 120 according to the second embodiment is configured not to perform a shutter operation until device data can be correctly acquired after a photographing start operation is performed.

[0233] Therefore, according to the second embodiment, the maintenance worker can reliably eliminate the need for re-shooting.

[0234] [Other embodiments]

[0235] In the first and second embodiments described above, the shutter operation is assigned as the operation for finalizing the character recognition result. However, an operation other than the shutter operation may be assigned as the operation for finalizing the character recognition result.

[0236] In the first and second embodiments, the shooting of the moving image data is terminated and the character recognition result is determined by performing a shutter operation. However, it is also possible to configure the shooting of the moving image data to be terminated and the character recognition result to be determined automatically under the condition that the character recognition result for the target item is stored in the first storage unit 425.

[0237] In addition, in the first and second embodiments described above, when the text recognition results are obtained for all the object items, the text recognition results are stored in the first storage unit 425. However, the method of storing in the first storage unit 425 is not limited to this. For example, as in the first and second embodiments described above, when a plurality of object items are included and a text recognition result is obtained for any of the object items, the obtained text recognition result is overwritten as the text recognition result of the corresponding object item. In other words, the text recognition results may be stored individually for each object item.

[0238] In addition, in the first and second embodiments described above, the case of reading device data from a device nameplate is described, but the object of reading device data is not limited to the device nameplate. Alternatively, the object read from the device nameplate is not limited to the device data. In addition, the object photographed by the camera is not limited to the device nameplate, and may be an object other than the device nameplate as long as it contains a specific text string.

[0239] In addition, in the first and second embodiments, the device data rules are exemplified when the field device is an outdoor unit of an air conditioner, but when the field devices are different, the device data rules corresponding to the different field devices are used. Similarly, in the first and second embodiments, the nameplate is exemplified when the field device is an outdoor unit of an air conditioner, but when the field devices are different, the nameplate corresponding to the different field devices is used.

[0240] In addition, in the first and second embodiments described above, for convenience of explanation, the case where the character recognition result obtained from the frame image is displayed is overlapped and displayed on the displayed frame image. However, since it takes a certain amount of time to obtain the character recognition result and overlap it, the frame image from which the character recognition result is obtained and the frame image from which the character recognition result is overlapped and obtained may not be consistent.

[0241] Although embodiments have been described above, it will be understood that various changes in form and details may be made without departing from the spirit and scope of the claims.

[0242] This application claims priority based on Japanese Patent Application No. 2022-140707 filed on September 5, 2022, the entire contents of which are incorporated herein by reference.

[0243] Description of Reference Numerals

[0244] 110: Field equipment

[0245] 111: Equipment nameplate

[0246] 120: Information Terminal

[0247] 121: Capture image

[0248] 305: Camera

[0249] 401: Operation receiving unit

[0250] 402: Get Department

[0251] 403: Identification

[0252] 404: Correction

[0253] 405: Extraction Department

[0254] 406: Judgment Department

[0255] 407: Border generation unit

[0256] 408: First storage control unit

[0257] 409: Second storage control unit

[0258] 410: Display unit

[0259] 411: Manual Correction Department

[0260] 1201: Operation receiving unit.

Claims

1. An information terminal having a control unit, wherein: The control unit: Repeating the process of acquiring the captured frame image and storing the character recognition result obtained by performing character recognition on the frame image in the storage unit, When a predetermined operation on the information terminal is accepted, a character recognition result is determined.

2. The information terminal according to claim 1, wherein: When the captured frame image is displayed in real time, a character recognition result obtained by performing the character recognition is displayed in an overlapping manner.

3. The information terminal according to claim 2, wherein: The control unit: At a first time interval, the frame image to be displayed is updated, The character recognition result displayed in an overlapping manner is updated at a second time interval that is longer than the first time interval.

4. The information terminal according to any one of claims 1 to 3, wherein: The control unit: When the captured frame image includes a plurality of items to be stored as text recognition results, the text recognition results of the items obtained by performing the text recognition are stored in the storage unit in units of items. When a predetermined operation on the information terminal is accepted, the character recognition result is determined in units of items.

5. The information terminal according to any one of claims 1 to 4, wherein: The control unit: When the displayed character recognition result is corrected in accordance with the determined character recognition result being displayed on the display unit, the corrected character recognition result is stored in the storage unit.

6. The information terminal according to any one of claims 1 to 5, wherein: The control unit: By performing text recognition on a frame image of a nameplate of a device, a model name and a production number are obtained as text recognition results.

7. The information terminal according to any one of claims 1 to 6, wherein: The control unit: storing the character recognition result obtained by performing the character recognition in the first storage unit, The character recognition result determined by accepting a predetermined operation on the information terminal is stored in the second storage unit.

8. The information terminal according to claim 7, wherein: The control unit: Whenever a character recognition result is obtained by performing the character recognition, the character recognition result stored in the first storage unit is updated.

9. The information terminal according to any one of claims 1 to 8, wherein: The character recognition result is a result determined by the control unit to satisfy a storage condition among the results obtained by performing the character recognition.

10. The information terminal according to claim 9, wherein: The control unit: When a result obtained for a predetermined item by performing the character recognition is a preset character string, it is determined that the storage condition is satisfied, and the result obtained for the predetermined item is stored in the storage unit as a character recognition result for the predetermined item.

11. The information terminal according to claim 7, wherein: The control unit: The frame image for which the character recognition result stored in the first storage unit is obtained is stored in the first storage unit in association with the character recognition result. The frame image for which the character recognition result stored in the second storage unit is obtained is stored in the second storage unit in association with the character recognition result.

12. A method for character recognition, wherein: The control unit of the information terminal executes the following steps: Repeating the process of acquiring the captured frame image and storing the character recognition result obtained by performing character recognition on the frame image in the storage unit; as well as When a predetermined operation on the information terminal is accepted, a character recognition result is determined.

13. A program for causing a control unit of an information terminal to execute the following steps: Repeating the process of acquiring the captured frame image and storing the character recognition result obtained by performing character recognition on the frame image in the storage unit; and When a predetermined operation on the information terminal is accepted, a character recognition result is determined.

Images