Digitalization methods, programs, and electronic devices

Abstract

This invention provides a digitization method that can digitize the entire scrolling text with excellent accuracy. [Solution] A digitization method for generating text data of a scroll string from multiple images, comprising the steps of: assigning sequential numbers to multiple characters constituting text data A obtained by converting string A, which is part of a scroll string contained in image A, in the order of the characters; assigning sequential numbers to multiple characters constituting text data B obtained by converting string B, which is part of a scroll string contained in image B, in the order of the characters constituting text data B, so that characters that have high commonality with text data A or text data generated based on text data A are assigned the same number; and generating integrated text data arranged in the order of the sequential numbers based on text data A and text data B.

Description

Technical Field

[0001] The present invention relates to a digitization method, a program, and an electronic device.

Background Art

[0002] There are known display units of devices that display scroll strings flowing in the horizontal or vertical direction, electronic bulletin boards, electronic signs, etc. (see, for example, Patent Document 1). By scrolling the string, it becomes possible to display a string longer than the width or height of the display unit or the like on the display unit or the like. Also, there are known information processing devices that digitize the strings included in images and output them as audio (see, for example, Patent Document 2), and translation devices that digitize the strings included in images captured by cameras, translate them into a predetermined language, and display them (see, for example, Patent Document 3).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0004] Since the string being scrolled on the display unit is longer than the width or height of the display unit or the like, the entire string is not displayed on the display unit at the same time. Therefore, it is difficult for conventional devices to digitize the entire scrolled string and output it as audio or translate it.Furthermore, in conventional devices, when character recognition errors occur, it is difficult to incorrectly digitize the strings included in images and accurately output them as audio or translate them. This invention has been made in view of these circumstances, and provides a digitization method that can digitize the entire scrollable text with excellent accuracy. [Means for solving the problem]

[0005] The present invention provides a digitization method for generating text data of a scroll string from a plurality of images having a portion of the scroll string. The digitization method of the present invention includes the steps of: assigning sequential numbers to a plurality of characters (represented by character codes) that constitute text data A obtained by converting a string A which is a portion of the scroll string contained in image A, in the order of their corresponding numbers; assigning sequential numbers to a plurality of characters (represented by character codes) that constitute text data B obtained by converting a string B which is a portion of the scroll string contained in image B, in the order of their corresponding numbers, so that characters that have high commonality with text data A or text data generated based on text data A are assigned the same number; and generating integrated text data arranged in the order of the sequential numbers based on text data A and text data B.

[0006] The present invention provides a program designed to cause a computer to execute the digitization method of the present invention. The present invention provides an electronic device equipped with a speaker that outputs the integrated text data generated using the digitization method of the present invention as sound. The present invention also provides an electronic device comprising at least one of the following: a screen provided for displaying translated text data in another language, which is a translation of the integrated text data generated using the digitization method of the present invention, as a string of characters; and a speaker provided for outputting the translated text data as sound. [Effects of the Invention]

[0007] According to the present invention, the entire scrollable text can be digitized with excellent accuracy. [Brief explanation of the drawing]

[0008] [Figure 1] This is a flowchart illustrating the digitalization method for one embodiment of the present invention. [Figure 2] This is a flowchart illustrating the digitalization method for one embodiment of the present invention. [Figure 3] Block diagram showing the configuration of an electronic device in one embodiment of the present invention. [Figure 4] (a) to (e) are the first to fifth images, respectively. [Figure 5] (a) to (g) are diagrams illustrating the method for digitizing the scrolling text contained in the first to fifth images. [Modes for carrying out the invention]

[0009] One embodiment of the present invention will be described below with reference to the drawings. The configurations shown in the drawings and the following description are illustrative, and the scope of the present invention is not limited to those shown in the drawings and the following description.

[0010] Figures 1 and 2 are flowcharts of the digitization method of this embodiment, and Figure 3 is a block diagram showing the configuration of an electronic device capable of implementing at least a part of the digitization method of this embodiment. The digitization method of this embodiment is a method for generating text data of a scroll string from a plurality of images having a portion of the scroll string. The digitization method of the present invention includes the steps of: assigning sequential numbers to a plurality of characters constituting text data A obtained by converting a string A which is a portion of the scroll string contained in image A, in the order of the characters; assigning sequential numbers to a plurality of characters constituting text data B obtained by converting a string B which is a portion of the scroll string contained in image B, in the order of the characters constituting text data B, so that characters that have a high degree of commonality with text data A or text data generated based on text data A are assigned the same number; and generating integrated text data arranged in the order of the sequential numbers based on text data A and text data B.

[0011] Furthermore, the digitization method of this embodiment may further include the step of assigning sequential numbers to a plurality of characters constituting the text data C in order, so that characters that have high commonality with the text data A and the text data B, or the text data generated based on the text data A and the text data B, are assigned the same number among the characters in the text data C obtained by converting the string C, which is part of the scroll string contained in the image C. In this case, the integrated text data is generated based on the text data A, the text data B, and the text data C.

[0012] Scrolling text refers to a string of text that scrolls across a device's display, electronic billboard, digital sign, computer screen, etc. (hereinafter referred to as "display"). Scrolling text may scroll from bottom to top or from right to left. The entire scrolling text is not displayed on the display; initially, only the beginning and part of the scrolling text are displayed, and the scrolling text moves across the display over time. Specifically, the first characters of the scrolling text scroll out sequentially, and subsequent characters scroll in sequentially. Scrolling text may be a sentence, a phrase, or a word. The characters that make up the scrolling text are characters that can be digitized using character codes (alphabet, kanji, hiragana, etc.), numbers, codes, or symbols. Text data is data that represents a string of characters according to a character code, and is digital data that can be used by computers and communications.

[0013] The electronic device 10 is not particularly limited as long as it is capable of implementing at least a part of the digitization method of this embodiment, but examples include smartphones, mobile phones, electronic devices for assisting the visually impaired, translation devices, personal computers, and network servers. The digitization method of this embodiment may be implemented jointly by these electronic devices. Furthermore, these electronic devices 10 are included in a computer. The electronic device 10 may include a control unit 2 having a memory unit 3, a camera 4, a screen 5, a speaker 6, a communication unit 7, an operation unit, and the like. The control unit 2 is the part that controls the electronic device 10. The control unit 2 may include, for example, an arithmetic processing unit (e.g., CPU), a memory unit 3 (e.g., RAM, storage device, etc.), a network controller, a video controller, a power supply unit, etc. The control unit 2 may also have multiple electronic circuits. The memory unit 3 may store a program or application that can implement the digitization method of this embodiment. The electronic device 10 may also be connected to a server 8 on the Internet network via a communication unit 7.

[0014] Figures 4(a) to 4(e) show the display area for the first to fourth menu items. Moving the cursor, indicated by ">", to the left edge of the display area and selecting a menu item will take you to the screen for that menu item. Also, for the third and fourth menu items, the entire string is not displayed on the display area; when the cursor is moved over the menu item, the string will scroll to be displayed. Figure 4(a) is the image immediately after the cursor is moved over the third menu item, which is displayed as "apple_banana_orange". In Figure 4(b), the first two characters of the third menu item scroll out and the next two characters scroll in, displaying "ple_banana_orange_d". In Figure 4(c), the next two characters of the third menu item scroll out and the next two characters scroll in, displaying "e_banana_orange_dog". In Figure 4(d), the next two characters of the third menu item scroll out and the next two characters scroll in, displaying "banana_orange_dog_c". In Figure 4(e), the last two characters of the third menu item scroll out and the next two characters scroll in, displaying "nana_orange_dog_cat". In this way, by scrolling the string, the entire scrollable string "apple_banana_orange_dog_cat" can be displayed.

[0015] The method for digitizing such scrolling text will be explained using the flowcharts shown in Figures 1 and 2. First, in step S1, a scroll string is photographed to obtain the first to the nth images. The number n of images is not particularly limited, but for example, it is 3 or more and 50 or less. For example, a still image or a moving image can be photographed using the camera 4 of the electronic device 10 to obtain an image. Also, an image may be obtained by capturing a screen output such as a personal computer. In the case of a moving image, an image may be obtained by extracting an image from the moving image. The first to the nth images are images obtained by photographing the scroll string in order. For example, they may be images obtained each time 1 or more and 4 or less characters scroll in. Also, image A, image B, and image C are images included in the first to the nth images. Image B is an image obtained later than image A, and image C is an image obtained later than image B. Here, FIG. 4(a) is used as the first image, FIG. 4(b) is used as the second image, FIG. 4(c) is used as the third image, FIG. 4(d) is used as the fourth image, and FIG. 4(e) is used as the fifth image. Image A may be the first image, the second image, or the third image. For example, when the first image is image A, the second image can be image B, and the third image can be image C. Also, step S1 may be performed in parallel with steps S2 to S9. Specifically, after obtaining the first image in step S1, steps S2 and S3 may be performed before obtaining the subsequent images. Also, after obtaining the second image in step S1, steps S2, S4 to S6 may be performed before obtaining the subsequent images. Also, after obtaining the images after the third image in step S1, steps S2, S7 to S9 may be performed before obtaining the subsequent images. Also, steps S1 to S3 may be performed for the first image, and then steps S1, S2, S4, S5, S6 may be performed for the second image, and then steps S1, S2, S7, S8, S9 may be performed for the third image. Steps S1, S2, S7, S8, S9 may be similarly performed in order for the images numbered 4 or higher.

[0016] In step S2, the character strings included in the first to nth images are respectively converted into the first to nth text data. For this conversion, an optical character recognition (OCR) technique can be used. For example, the control unit 2 of the electronic device 10 may perform optical character recognition on the character string included in the image to generate text data. Also, steps S2 to S9 may be performed by the control unit 2 of the electronic device 10. Further, the control unit 2 of the electronic device 10 may transmit the image to the server 8 on the Internet network via the communication unit 7, and receive the text data generated by performing optical character recognition on the character string included in the image by the server 8 via the communication unit 7. Also, steps S2 to S9 may be performed by the server 8, and the control unit 2 of the electronic device 10 may receive the generated integrated text data via the communication unit 7. Here, the same number is assigned to the image and the text data. Specifically, the data generated by performing optical character recognition on the character string included in the first image is the first text data, and the data generated by performing optical character recognition on the character string included in the second image is the second text data. The same applies to other numbers. Also, the data generated by performing optical character recognition on the character string included in image A is text data A, the data generated by performing optical character recognition on the character string included in image B is text data B, and the data generated by performing optical character recognition on the character string included in image C is text data C.

[0017] In step S3, consecutive numbers are assigned to the plurality of characters constituting the first text data. For example, consecutive numbers (1 to 19) starting from 1 as shown in FIG. 5(a) are assigned in order to the plurality of characters constituting the first text data "apple_banana_orange" generated by performing optical character recognition on the character string of the third menu item included in the first image shown in FIG. 4(a). In this specification and drawings, the text data is described as the characters it represents.

[0018] In step S4, temporary sequential numbers are assigned to multiple characters that make up the second text data, and the number of characters that make up the first text data that have the same number as the character that makes up the second text data when each temporary sequential number is assigned is counted.

[0019] For example, first, the string of the third menu item contained in the second image shown in Figure 4(b) is optically recognized to obtain the second text data "ple_banana_oran9e_d". The characters constituting this second text data are assigned temporary sequential numbers (1-19) starting from 1, as shown in Figure 5(b), in the order they appear. In this second text data, "g" is incorrectly converted to "9" in step S2 (character recognition error). Then, for each sequential number, the first text data and the second text data are compared, and the number of identical characters is counted. In Figure 5(b), the characters are the same at numbers 8, 9, and 10, resulting in a total of 3 identical characters.

[0020] Next, we assign temporary sequential numbers (2-20) to the characters that make up the second text data "ple_banana_oran9e_d" in the order they appear, starting from 2 as shown in Figure 5(c). Then, we compare the first text data and the second text data for each sequential number and count the number of identical characters. In Figure 5(c), the characters are the same at number 2, so the number of identical characters is 1. Next, we assign temporary sequential numbers (3-21) starting from 3 to each character that makes up the second text data "ple_banana_oran9e_d" in the order shown in Figure 5(d). Then, we compare the first text data and the second text data for each sequential number and count the number of identical characters. In Figure 5(d), the same characters are found in numbers 3-17 and 19, resulting in a total of 16 identical characters. In this way, the first and second text data are compared while increasing the initial number of the temporary sequence number, and the number of identical characters when each temporary sequence number is assigned is counted. For example, in the example in Figure 5, the initial number of the temporary sequence number can be 1 to 19. However, if there is a temporary sequence number with a large number of identical characters, the comparison between the first and second text data can be stopped at that stage. For example, if more than 60% of the characters that make up the second text data are the same as the characters that make up the first text data at that number, the comparison between the first and second text data can be stopped. The same applies when comparing the third text data with the first integrated text data, when comparing the A text data with the (A-2) integrated text data, when comparing the third text data with the first and second text data, and when comparing the nth text data with the first to (n-1) text data.

[0021] In step S5, the provisional sequential number with the most identical characters is determined as the sequential number for multiple characters that make up the second text data. In the example in Figure 5, the provisional sequential number starting from 3 (3-21) as shown in Figure 5(d) has the most characters, so this provisional sequential number is determined as the sequential number for multiple characters that make up the second text data. By determining the sequential number in this way, even if character recognition errors occur for some characters in step S2, it is possible to suppress the misassignment of sequential numbers.

[0022] In step S6, first integrated text data is generated, arranged in sequential order based on the first and second text data. In step S6, if all the numbered characters are the same or if there is only one numbered character, that character can be used as the character with that number in the first unified text data. If the numbered characters are different, the character in the first text data can be used as the character with that number in the first unified text data. For example, in Figure 5(d), the characters numbered 1, 2, 20, and 21 are all the same character, and these characters can be used as the corresponding characters in the first unified text data. Also, the characters numbered 3 through 17 and 19 are the same character for each number, so these characters can be used as the corresponding characters in the first unified text data. The character numbered 18 is "g" in the first text data and "9" in the second text data, so "g" can be used as the corresponding character in the first unified text data. Therefore, the first unified text data is "apple_banana_orange_d".

[0023] In step S7, temporary sequential numbers are assigned to multiple characters that make up the third text data, and the number of characters that make up the first integrated text data that have the same number as the character that makes up the third text data when each temporary sequential number is assigned is counted. Step S7 is the same as step S4 described above, but for the third text data. However, in step S4, the first text data and the second text data are compared and the same characters are counted, whereas in step S7, the third text data and the first integrated text data are compared and the same number is counted. Alternatively, the third text data may be compared with the first and second text data to count the same number.

[0024] For example, in Figure 5(e), the characters that make up the third text data "e_banana_orange_dog" are assigned temporary sequential numbers (1-19) starting from 1, according to their order. In this case, at numbers 8 and 12, the characters in the third text data are the same as the characters in the first integrated text data, and the number of identical characters is 2.

[0025] For example, in Figure 5(f), the characters that make up the third text data "e_banana_orange_dog" are assigned temporary sequential numbers (5-23) starting from 5, according to their order. In this case, the characters in the third text data from 5 to 21 are the same as those in the first unified text data. Therefore, the number of identical characters is 17.

[0026] In step S8, the provisional sequential numbers with the most identical characters are confirmed as the sequential numbers of multiple characters that make up the third text data. In the example in Figure 5, the provisional sequential numbers starting from 5 (5~23) as shown in Figure 5(f) have the most characters, so these provisional sequential numbers are confirmed as the sequential numbers of multiple characters that make up the third text data.

[0027] In step S9, a second unified text data set is generated, arranged in sequential order based on the first, second, and third text data sets. In step S9, if all the numbered characters are the same or if there is only one numbered character, that character can be used as the character corresponding to that number in the second integrated text data. For example, in Figure 5(f), there is only one character assigned the numbers 1, 2, 22, and 23, and that character can be used as the character corresponding to that number in the second integrated text data. Also, the characters assigned the numbers 3-17 and 19-21 are the same character for each number, so those characters can be used as the characters corresponding to that number in the second integrated text data.

[0028] In step S9, if three or more characters assigned the same number contain different characters, the most frequent character among those three or more characters can be used as the character for that number when generating the integrated text data. In other words, the character for that number can be determined by majority vote (voting method). For example, in Figure 5(f), number 18 contains two "g"s and one "9". Therefore, "g" can be used as the character for number 19 in the second integrated text data. The "9" in number 19 is a character recognition error in step S2, but by determining the character for that number by majority vote (voting method), it is possible to prevent characters with character recognition errors from being included in the second integrated text data. In the example shown in Figure 5(f), performing step S9 as described above allows for the accurate generation of a second unified text data set named "apple_banana_orange_dog" arranged in sequential order from 1 to 23.

[0029] Steps S7, S8, and S9 are performed in the same manner for the 4th to nth text data. In the examples in Figures 4 and 5, steps S7, S8, and S9 are also performed in order for the 4th and 5th text data. Furthermore, as shown in Figure 5(g), in step S8 for the fourth text data, a provisional sequence number starting from 7 (7-25) is determined as the sequence number of multiple characters constituting the fourth text data, and in step S9, the third integrated text data "apple_banana_orange_dog_c" arranged in the order of sequence numbers 1-25 can be accurately generated. The hyphen "-" in the number 13 of the fourth text data is a character recognition error in step S2, but by determining the character for that number by majority vote (voting method), it is possible to suppress the inclusion of characters with character recognition errors in the third integrated text data. In step S8 for the fifth text data, a provisional sequence number starting from 9 (9-27) is determined as the sequence number of multiple characters that make up the fifth text data, and in step S9, the fourth integrated text data "apple_banana_orange_dog_cat" can be accurately generated with the sequence numbers 1-27 in order.

[0030] As shown in Figure 5, in the case of scrollable strings, the sequential number gradually increases as the steps described above are followed. However, in the case of non-scrollable strings, the sequential number does not change even if the steps described above are followed. Therefore, processing can be interrupted for strings whose sequential number does not change. For example, the strings "Main" for the first menu item, "test.gcode" for the second menu item, and "Small_Pizza_Planet_" for the fourth menu item in the image shown in Figure 4 are not scrollable. Therefore, after confirming that the sequential number does not change by repeating steps S4 and S5 several times, the repetition of steps S7, S8, and S9 for the text data of these strings can be stopped, and the text data can be finalized.

[0031] In step S10, the control unit 2 of the electronic device 10 determines whether or not to output the integrated text data as audio. For example, the control unit 2 can determine whether or not to output the audio based on the user's operation of the control unit of the electronic device 10 or the settings of the electronic device 10. If the control unit 2 determines to output the audio, it outputs the integrated text data as audio using the speaker 6 (step S11). The integrated text data in steps S10 and S11 may be the final integrated text data (the fourth integrated text data in the example of Figures 4 and 5), or it may be intermediate integrated text data (the first integrated text data, the second integrated text data, or the third integrated text data in the example of Figures 4 and 5). Furthermore, the control unit 2 can also determine whether or not to output the text data of a string that has not been scrolled as audio. If it determines that audio output is appropriate, it will output the text data as audio using the speaker 6.

[0032] In step S12, the control unit 2 of the electronic device 10 determines whether or not to translate the integrated text data into another language. For example, the control unit 2 can determine whether or not to translate based on the user's operation of the control unit of the electronic device 10 or the settings of the electronic device 10. If it is determined to translate, the control unit 2 translates the integrated text data and generates translated text data (step S13). This translation may be performed using programs and data stored in the storage unit 3. Alternatively, the control unit 2 may transmit the integrated text data to the server 8 via the communication unit 7, and the control unit 2 may receive the translated text data, which has been translated from the integrated text data by the server 8, via the communication unit 7. The integrated text data in steps S12 and S13 may be the final integrated text data (the fourth integrated text data in the examples of Figures 4 and 5), or it may be intermediate integrated text data (the first integrated text data, the second integrated text data, or the third integrated text data in the examples of Figures 4 and 5). Furthermore, the control unit 2 can also determine whether or not to translate text data of strings that are not being scrolled. If it determines that translation is necessary, the control unit 2 or server 8 translates the text data and generates translated text data.

[0033] In step S14, the control unit 2 of the electronic device 10 determines whether or not to output the translated text data as audio. For example, the control unit 2 can determine whether or not to output the audio based on the user's operation of the control unit of the electronic device 10 or the settings of the electronic device 10. If the control unit 2 determines to output the audio, it outputs the translated text data as audio using the speaker 6 (step S15).

[0034] In step S16, the control unit 2 of the electronic device 10 determines whether or not to display the translated text data on the screen 5. For example, the control unit 2 can determine whether or not to display it on the screen 5 based on the user's operation of the control unit of the electronic device 10 or the settings of the electronic device 10. If the control unit 2 determines that it should display the data, it displays the translated text on the screen 5 using the translated text data (step S17). [Explanation of Symbols]

[0035] 2: Control Unit 3: Memory Unit 4: Camera 5: Screen 6: Speaker 7: Communication Unit 8: Server 10: Electronic Devices

Claims

1. A digitization method for generating text data of a scroll string from multiple images having a portion of the scroll string, The steps include: assigning sequential numbers to multiple characters that constitute text data A obtained by converting string A, which is part of the scroll string contained in image A, in the order of their characters; The steps include assigning sequential numbers to multiple characters constituting the text data B in order so that characters in the text data B obtained by converting the string B which is part of the scroll string contained in image B have high commonality with the text data A or the text data generated based on the text data A, and assigning the same number to each character in the text data B. A digitization method comprising the step of generating integrated text data arranged in sequential order based on the text data A and the text data B.

2. The process further includes the step of assigning sequential numbers to a plurality of characters constituting the text data C in order, so that characters in the text data C obtained by converting the string C which is part of the scroll string contained in image C have high commonality with the text data A and the text data B or the text data generated based on the text data A and the text data B, among the characters contained in the text data C, which are the same characters. The digitization method according to claim 1, wherein the integrated text data is generated based on the text data A, the text data B, and the text data C.

3. The digitization method according to claim 2, which generates the integrated text data by using the most frequent character among three or more characters assigned the same number as the character of that number.

4. A program provided for causing a computer to execute the digitization method described in any one of claims 1 to 3.

5. An electronic device comprising a control unit and a speaker, which are provided to output the integrated text data generated using the digitization method described in any one of claims 1 to 3 as audio.

6. An electronic device comprising at least one of the following: a screen provided for displaying translated text data in another language, obtained by translating the integrated text data generated using the digitization method described in any one of claims 1 to 3, as a string; and a speaker provided for outputting the translated text data as sound.

Images