Screen data processing apparatus, method, and program

Abstract

A screen data processing device according to an embodiment comprises: a detection degree calculation unit which, on the basis of both screen component information that manages information about character strings that are components of screen data, and a partial region of an image displayed on a display screen, calculates a character string detection degree indicating the degree to which the character strings managed by the screen component information are detected to be drawn in the region; and a determination unit which determines whether or not the character strings managed by the screen component information are drawn in the region, on the basis of the magnitude of the character string detection degree calculated by the detection degree calculation unit.

Description

Technical Field

[0001] Embodiments of the present invention relate to a screen data processing apparatus, method, and program.

Background Art

[0002] When a counselor who is a business operator conducts business, text information displayed on a terminal screen is acquired and accumulated, and based on this information, when a problem occurs, the counselor and a candidate counselor who is a business operator who can smoothly solve this problem are paired. A technology, so-called counselor matching technology, has been proposed (see, for example, Non-Patent Document 1).

[0003] Generally, when a candidate counselor responds to a consultation based on a solution case experienced in the past, for example, the candidate searches for his or her own past documents or messages on a communication tool, or redisplay the past cases on the business system screen.

[0004] In the counselor matching technology, as a method of reducing the necessity of the above work by the candidate counselor, for example, in addition to the text information displayed on the screen when using the terminal, the screen image and the path of the document are acquired, and they are associated, recorded, and accumulated. Then, when the candidate counselor accepts a consultation, the accumulated information is presented to the candidate counselor.

Prior Art Documents

Non-Patent Documents

[0005]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] Inconsistencies may occur between the screen image obtained for the candidate for consultation as described above, the text information obtained at the same timing as the screen image, and the information of the screen component object which is the extraction source of the text information.

[0007] The triggers for the occurrence of such inconsistencies include, for example, tab switching on the screen, screen transition, reduction and expansion of the panel, and dynamic addition or deletion of screen components, etc. When the above-mentioned inconsistency occurs, the candidate for consultation cannot understand and utilize the cases experienced in the past from the screen image where the above-mentioned inconsistency occurred.

[0008] In addition, there is a risk of confusing the candidate for consultation by presenting a screen image that has nothing to do with the text information.

[0009] One of the causes of the above-mentioned inconsistency is, for example, a deviation in the screen display timing. For example, the timing at which the application program that is the target of screen data acquisition creates and updates the information of the screen component object is different from the timing at which the screen image is drawn.

[0010] Another cause is the deviation between the screen display timing and the screen acquisition timing. For example, the timing at which the program that acquires the screen data acquires the information of the screen component object and the screen image is not consistent with the timing of the above-mentioned screen display.

[0011] The context and time lag before and after the above screen display timing depend on the GUI (Graphical User Interface) platform, version, and load status of the operating environment used for the screen display. Therefore, there are limitations to a uniform approach by a general-purpose screen data acquisition program. Also, blocking screen operations to acquire data at a timing when the screen does not change is inappropriate because it hinders the terminal operations of the candidate counselors.

[0012] The present invention has been made in view of the above circumstances, and an object thereof is to provide a screen data processing apparatus, method, and program capable of determining whether an image displayed on a display screen matches a component managed by screen component information.

Means for Solving the Problems

[0013] A screen data processing apparatus according to an aspect of the present invention manages information of a character string that is a component of screen data Obtained together with the image from the display screen as screen component information, The and based on a partial area of an image displayed on a display screen, a detection degree calculation unit that calculates a character string detection degree indicating how much the character string managed by the screen component information is drawn in the area, and the character string detection degree calculated by the detection degree calculation unit Comparison with the threshold value Based on this, a determination unit that determines whether or not the character string managed by the screen component information is drawn in the area is provided.

[0014] A screen data processing method according to an aspect of the present invention is a method performed by a screen data processing apparatus Wa and includes: By the detection degree calculation unit of the screen data processing device managing information of a character string that is a component of screen data Obtained together with the image from the display screen as screen component information, The Based on a partial area of an image displayed on a display screen, calculating a character string detection degree indicating how much the character string managed by the screen component information is drawn in the area Thing and By the determination unit of the screen data processing device Based on the string detection degree calculated by the detection degree calculation unit Comparison with the threshold value determine whether or not a string managed by the screen component information is drawn in the area Thing and includes.

Effect of the Invention

[0015] According to the present invention, it is possible to determine whether or not an image displayed on a display screen matches a component managed by screen component information.

Brief Description of the Drawings

[0016] [Figure 1] FIG. 1 is a diagram showing an application example of a screen data processing apparatus according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram showing an example of details of the function of the screen data consistency inspection unit of the screen data processing apparatus. [Figure 3] FIG. 3 is a diagram showing an example of details of the function of the screen data display unit of the screen data processing apparatus. [Figure 4] FIG. 4 is a diagram showing an example of the configuration of information of a screen component object in tabular form. [Figure 5] FIG. 5 is a diagram showing an example of the relationship between each of a plurality of screen component objects of information of a screen component object. [Figure 6] FIG. 6 is a diagram showing an example of a screen image. [Figure 7] FIG. 7 is a diagram showing an example of the attributes of a screen in tabular form. [Figure 8] FIG. 8 is a flowchart showing an example of the processing operation of the screen data processing apparatus. [Figure 9] FIG. 9 is a diagram showing an example of the inspection of screen data consistency by the screen data processing apparatus. [Figure 10] FIG. 10 is a diagram showing an example of determination of the presence or absence of a display string using a threshold value. [Figure 11] FIG. 11 is a diagram showing an example of determination of the presence or absence of a display string using a reference at the time of inconsistency and a reference at the time of consistency. [Figure 12] FIG. 12 is a flowchart showing an example of a processing operation related to calculation of a reference at the time of inconsistency by a screen data processing apparatus. [Figure 13] FIG. 13 is a diagram showing an example of calculation of a reference at the time of inconsistency by a screen data processing apparatus. [Figure 14] FIG. 14 is a flowchart showing an example of a processing operation related to calculation of a reference at the time of consistency by a screen data processing apparatus. [Figure 15] FIG. 15 is a diagram showing an example of calculation of a reference at the time of consistency by a screen data processing apparatus. [Figure 16] FIG. 16 is a flowchart showing an example of a processing operation related to calculation of a string detection degree reduction rate by a screen data processing apparatus. [Figure 17] FIG. 17 is a diagram showing an example of a string detection degree calculated for a pseudo-consistent image. [Figure 18] FIG. 18 is a diagram showing an example of a reference at the time of consistency calculated according to a string detection degree reduction rate. [Figure 19] FIG. 19 is a diagram showing an example of collection of consistent images. [Figure 20] FIG. 20 is a block diagram showing an example of a hardware configuration of a screen data processing apparatus according to an embodiment of the present invention. MODE FOR CARRYING OUT THE INVENTION

[0017] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a diagram showing an application example of a screen data processing apparatus according to an embodiment of the present invention. As shown in FIG. 1, a screen data processing apparatus 100 according to an embodiment of the present invention includes an input unit 11, a screen data extraction unit 12, a screen data consistency inspection unit 13, a screen data exclusion or selection unit 14, a screen data holding unit 15, a screen data display unit 16, a detection degree reduction rate calculation unit 17, a detection degree reduction rate holding unit 18, a consistent image holding unit 19, and a screen data consistency degree holding unit 20.

[0018] FIG. 2 is a diagram showing an example of details of the function of the screen data consistency inspection unit of the screen data processing apparatus. As shown in FIG. 2, the screen data consistency inspection unit 13 of the screen data processing apparatus 100 includes an inspection target area selection unit 13-1, a pseudo-inconsistent image generation unit 13-2, an inconsistency criterion calculation unit 13-3, an inconsistency criterion holding unit 13-4, a pseudo-consistent image generation unit 13-5, a consistency criterion calculation unit 13-6, a consistency criterion holding unit 13-7, a display character string detection degree calculation unit 13-8, a display character string presence / absence determination unit 13-9, and a screen data consistency degree calculation unit 13-10.

[0019] FIG. 3 is a diagram showing an example of details of the function of the screen data display unit of the screen data processing apparatus. As shown in FIG. 3, the screen data display unit 16 of the screen data processing apparatus 100 includes a copy operation detection unit 16-1, a display character string replication unit 16-2, and a consistent image recording unit 16-3. Details of the functions of the respective units shown in FIGS. 1 to 3 will be described later.

[0020] FIG. 4 is a diagram showing, in tabular form, a configuration example of information on screen component objects. The screen component objects shown in FIG. 4 are information held in the screen data holding unit 15 in screen data related to a certain screen displayed in the past, and the screen component ID, type, display or non-display state, display character string, and the drawing area of the screen component in the screen image are shown for each of a plurality of screen component objects.

[0021] FIG. 5 is a diagram showing an example of the relationship between each of a plurality of screen component objects of information on screen component objects. In the example shown in FIG. 5, among the screen data that is the same as the screen data corresponding to the information shown in FIG. 4, the information held in the screen data holding unit 15, the relationships of each of the screen component objects corresponding to the screen component IDs “1” to “44” shown in FIG. 4 are shown in a tree format.

[0022] FIG. 6 is a diagram showing an example of a screen image. In the example shown in FIG. 6, a confirmation screen for the order details of an article in the same screen data as the screen data corresponding to the information shown in FIG. 4 is shown as a screen image. FIG. 7 is a diagram showing an example of screen attributes in a table format. In the example shown in FIG. 7, as attributes of the screen image, a title, a class name, coordinate values of a display area, and a program name are shown in the same screen data as the screen data corresponding to the information shown in FIG. 4. The various types of information shown in FIGS. 4 to 7 are information related to the same screen. In the present embodiment, a set of these information is referred to as screen data.

[0023] Next, an example of the processing operation of the screen data processing apparatus 100 will be described. First, the basic processing operation of the screen data processing apparatus 100 will be described. FIG. 8 is a flowchart showing an example of the processing operation of the screen data processing apparatus. FIG. 9 is a diagram showing an example of the inspection of screen data consistency by the screen data processing apparatus. The screen data processing apparatus 100 compares the information of the screen component object with the screen image and inspects whether they are consistent with each other.

[0024] For each screen component, the screen data processing apparatus 100 calculates a character string detection degree d indicating how much of the display character string obtained from the information of the screen component object can be detected within the screen image, and based on the comparison between this character string detection degree d and a threshold value d 0 determines whether the display character string is drawn or not, that is, whether the display character string is appropriately drawn on the screen image.

[0025] In addition, the screen data processing device 100 calculates the screen data integrity, which is the ratio of the screen components among the information of the screen component objects, for which it is determined that the display string is drawn in the screen image, and ranks or selects the screen data based on this screen data integrity.

[0026] In this processing operation, the information of the screen component object is compared with the screen image, and it is inspected whether they are in agreement. Through this inspection, a screen image that is not in agreement with the information of the screen component object can be excluded from being presented to the user, for example, a candidate for consultation. In addition, a combination of text information and a screen image that is in agreement with the information of the screen component object, or a combination with better agreement, can be selected and presented to the user.

[0027] The screen data holding unit 15 stores the information of the screen component object and the information of the screen image in the screen data regarding the screens displayed in the past. The screen data extraction unit 12 extracts the screen data held in the screen data holding unit 15 based on the extraction conditions of the screen data input by the input unit 11, and for each screen component in the information of the screen component object of this screen data, obtains the display / non-display state, the drawing area of the screen component, and the display string. The inspection target area selection unit 13-1 of the screen data integrity inspection unit 13 specifies the inspection target areas in the screen component object and the screen image according to the acquisition result (reference symbol x in FIG. 9) (S11).

[0028] Next, the display string detection degree calculation unit 13-8 calculates, for each screen component displayed in the screen image, the string detection degree d, that is, how much display string is detected from the drawing area of the screen component object (S12).

[0029] In the example shown in FIG. 9, three types of string inspection methods are shown, and the string detection degree d can be calculated by any of the inspection methods. In the first character string inspection method, after the specification in S11, a verification image in which the display character string "area classification" in the screen component object is drawn is generated (reference sign a in FIG. 9), an image of the area to be inspected is extracted from the screen image, and these verification images and the extracted image are collated by template matching, and the matching degree between the collated images is calculated as the character string detection degree d.

[0030] In the second character string inspection method, after the specification in S11, the display character string "area classification" in the screen component object is extracted (reference sign b in FIG. 9), an image of the area to be inspected is extracted from the screen image, and the display character string "area classification" and the character string indicated by the image of the area to be inspected are collated by OCV (Optical character verification), and it is verified whether the display character string "area classification" in the screen component object is drawn in the image of the area to be inspected. The success degree of this collation is calculated as the character string detection degree d.

[0031] In the third character string inspection method, after the specification in S11, the display character string "area classification" in the screen component object is extracted (reference sign c in FIG. 9), an image of the area to be inspected is extracted from the screen image, and the character string indicated by this image is read by OCR (Optical character recognition). Then, the read character string and the display character string "area classification" in the screen component object are collated, and the similarity between the two is calculated as the character string detection degree d.

[0032] FIG. 10 is a diagram showing an example of determination of the presence or absence of a display character string using a threshold value. The display character string presence / absence determination unit 13-9 compares the calculated character string detection degree with the threshold value d of the character string detection degree 0By comparing with (Fig. 10), it is determined for each of a plurality of screen components indicated by the information of the screen component object whether or not a display string is drawn appropriately on the screen image (S13). When the processing from S11 to S13 has not been completed for each of the plurality of screen components indicated by the information of the screen component objects in the same screen (No in S14), the process returns to S11, and the processing from S11 to S13 is performed for the screen component before processing among each of the plurality of screen components.

[0033] Fig. 10 is a diagram showing an example of determination of whether or not a display string is drawn based on the string detection degree. In the example shown in Fig. 10, the string detection degree d is calculated for each of three types of screen components managed by the information of the screen component object, here screen component A, screen component B, and screen component C.

[0034] And the string detection degree d calculated for screen component A is higher than the threshold value d 0 Therefore, for this screen component A, the determination result of whether or not a display string is drawn is "display string is drawn". The string detection degree d calculated for screen component B is lower than the threshold value d 0 Therefore, for this screen component C, the determination result of whether or not a display string is drawn is "display string is not drawn". The string detection degree d calculated for screen component C is higher than the threshold value d 0 Therefore, for this screen component C, the determination result of whether or not a display string is drawn is "display string is drawn". Note that in the present embodiment, when the calculated string detection degree d is the same as the threshold value d 0 the determination result of whether or not a display string is drawn is "display string is drawn".

[0035] When the processing from S11 to S13 is completed for each of the plurality of screen components indicated by the information of the screen component objects in the same screen (Yes in S14), the screen data integrity calculation unit 13-10 calculates, based on the determination result in S13, for each of the plurality of screen images, the screen data integrity, which is the ratio of the screen components determined in S13 to have the display string drawn, among the plurality of screen components indicated by the information of the screen component objects, and holds the calculation result in the screen data integrity holding unit 20 (S15).

[0036] Based on the screen data integrity calculated in S15 for each of the plurality of screen images, the screen data exclusion or selection unit 14 can exclude the screen images with the integrity below a certain level from the presentation targets to the user by the screen data display unit 16, or cause the screen data display unit 16 to display information in which a plurality of screen images are arranged according to the magnitude of the screen data integrity (S16).

[0037] This makes it possible not to display on the screen an image that has no relation to the display string of the screen component, and to eliminate confusion for the user caused by the presentation of irrelevant screen images.

[0038] Next, another example of the determination of the presence or absence of the display string drawing based on the string detection degree in the processing operation of the screen data processing apparatus 100 will be described. In the above, it has been described that the string detection degree is calculated using template matching, ОCV, or ОCR. However, due to factors such as the difference in the display string for each screen component, the difference in the background of the screen on which the display string is drawn for each screen component, or the difference in the language or character type setting in ОCR, the calculated string detection degree and the difficulty of detection vary. Therefore, in the method of determining the presence or absence of the display string using a certain threshold value d 0 even when the string in the screen image matches the display string in the information of the screen component object, it is not always surely determined.

[0039] Therefore, in another example of determining whether to draw a display string based on the string detection degree, according to the screen data and the display string to be inspected, a criterion for determination at the time of matching between the display string and the screen image, that is, the string detection degree d (which may also be simply referred to as the criterion at the time of matching), which is the criterion for the determination result in terms of whether to draw the display string to be "there is a display string". upper And a criterion for determination at the time of mismatch between the display string and the screen image, that is, the string detection degree d (which may also be simply referred to as the criterion at the time of mismatch), which is the criterion for the determination result in terms of whether to draw the display string to be "no display string". lower are dynamically generated and used instead of the above threshold d 0 By comparing these criteria with the above-calculated string detection degree, whether to draw the display string is determined.

[0040] FIG. 11 is a diagram showing an example of determining whether there is a display string using the criterion at the time of mismatch and the criterion at the time of matching. In the example shown in FIG. 11, for each of three types of screen components managed by the information of the screen component object, here, screen component A, screen component B, and screen component C, the string detection degree d is calculated. Further, the criterion at the time of matching d upper and the criterion at the time of mismatch d lower corresponding to screen component A are calculated, the criterion at the time of matching d upper and the criterion at the time of mismatch d lower corresponding to screen component B are calculated, and the criterion at the time of matching d upper and the criterion at the time of mismatch d lower corresponding to screen component C are calculated. Suppose.

[0041] The string detection degree d calculated for screen component A is between the criterion at the time of mismatch d lower and the criterion at the time of matching d upper calculated for screen component A, and is closer to the criterion at the time of mismatch d upper compared with the criterion at the time of matching d lower , that is, the criterion at the time of mismatch d lower and the criterion at the time of matching d upperSince it is lower than the average with [the relevant item], for this screen component A, the determination result regarding whether to draw the display string is "no display string drawing".

[0042] Also, the string detection degree d calculated for screen component B is higher than the reference value d at the time of consistency calculated for screen component B upper so, for this screen component A, the determination result regarding whether to draw the display string is "display string drawing exists".

[0043] The string detection degree d calculated for screen component C is between the reference value d at the time of inconsistency calculated for screen component C lower and the reference value d at the time of consistency upper and is closer to the reference value d at the time of consistency than the reference value d at the time of inconsistency lower i.e., higher than the average of the reference value d at the time of inconsistency upper and the reference value d at the time of consistency lower so, for this screen component C, the determination result regarding whether to draw the display string is "display string drawing exists". upper Through such determination, the difficulty of selecting the threshold value d

[0044] is eliminated so that errors in string detection determination due to differences in the above factors, and errors in string detection determination do not occur in situations where there are differences in the above factors. 0 As a result, fluctuations in the detection difficulty can also be reflected in the calculation result of the screen data consistency degree. This enables fluctuations in the detection difficulty to be reflected in the calculation result of the screen data consistency degree as well.

[0045] Next, details of the generation of the above reference value d at the time of inconsistency lower will be described. FIG. 12 is a flowchart showing an example of a processing operation related to the calculation of the reference value at the time of inconsistency by the screen data processing apparatus. FIG. 13 is a diagram showing an example of the calculation of the reference value at the time of inconsistency by the screen data processing apparatus. First, the pseudo-inconsistent image generation unit 13-2 selects one or more pieces of screen data that are obviously inconsistent from the screen data acquired by the screen data acquisition program (reference sign x in FIG. 13) (S21). For example, in S21, the screen data obtained at a timing when the application to be operated on the screen data to be inspected is not being executed is selected.

[0046] Then, the pseudo-inconsistent image generation unit 13-2 selects, as a pseudo-inconsistent region, one or more regions at an arbitrary position in the screen image of the screen data selected in S21, which have the same size as the drawing region of the screen component to be inspected and do not match the screen component, and extracts the image within this region as a pseudo-inconsistent image (S22).

[0047] The inconsistency criterion calculation unit 13-3 calculates the character string detection degree d - for the pseudo-inconsistent image generated in S22, and calculates this calculated character string detection degree d - as the criterion d lower for the character string detection degree at the time of inconsistency, and holds the calculation result in the inconsistency criterion holding unit 13-4 (S23). That is, "d lower =d - ".

[0048] When a plurality of pseudo-inconsistent images are extracted in S22, in order to eliminate as much as possible the influence of the accidental use of the region where the same character string as the display character string is drawn, for example, the median value of the character string detection degrees d - calculated for each pseudo-inconsistent image may be used.

[0049] In the example shown in FIG. 13, the character string detection degree d - can be calculated by any of the three character string inspection methods also shown in FIG. 9. In the first character string inspection method, after the generation of the pseudo-inconsistent image in S22, a reference image on which the display character string "area classification" in the screen component object is drawn is generated (reference numeral a in FIG. 13), and this reference image and the pseudo-inconsistent image (reference numeral y in FIG. 13) are collated by template matching, and the matching degree between the collated images is calculated as the character string detection degree d - .

[0050] In the second string inspection method, after generating the pseudo-inconsistent image in S22, the display string "Region Classification" in the screen component object is extracted (reference symbol b in FIG. 13), and this display string "Region Classification" and the string shown in the pseudo-inconsistent image are collated by OCV, and it is verified whether the display string "Region Classification" in the screen component object is drawn in the pseudo-inconsistent image. The success degree of this collation is the string detection degree d - is calculated as such.

[0051] In the third string inspection method, after generating the pseudo-inconsistent image in S22, the display string "Region Classification" in the screen component object is extracted (reference symbol c in FIG. 13), and the string shown in the pseudo-inconsistent image is read by OCR. Then, this read string and the display string "Region Classification" in the screen component object are collated, and the similarity between the two is the string detection degree d - is calculated as such.

[0052] Next, the details of the generation of the above-mentioned matching criterion d upper will be described. FIG. 14 is a flowchart showing an example of a processing operation related to the calculation of the matching criterion by the screen data processing device. FIG. 15 is a diagram showing an example of the calculation of the matching criterion by the screen data processing device. First, the pseudo-matching image generation unit 13-5 generates a pseudo-matching image from the information of the screen component object (reference symbols x1, x2 in FIG. 15) (S31). This pseudo-matching image has a drawing area of the same size as the drawing area of the screen component, and is an image corresponding to the area where the display string is drawn, that is, the area that matches the screen component. Also, a plurality of pseudo-matching images may be generated while changing the types of backgrounds or fonts (fonts) prepared in advance.

[0053] Next, the matching criterion calculation unit 13-6 calculates the string detection degree d + for the pseudo-matching image generated in S31, and the calculated string detection degree d+ is calculated as the reference d for the string detection degree at the time of alignment, and the calculation result is held in the alignment reference holding unit 13-7 (S33). That is, "d upper = d upper ". +

[0054] When a plurality of pseudo-alignment images are created in S32, the worst value, average value, or median value of the string detection degrees d + calculated for each pseudo-alignment image can be used in subsequent processing.

[0055] In the example shown in FIG. 15, the string detection degree d + can be calculated by any of the three string inspection methods shown in FIG. 9 etc. In the first string inspection method, after generating the pseudo-alignment image in S32, a verification image in which the display string "area classification" in the screen component object is drawn is generated (reference sign a in FIG. 15), and this verification image and the pseudo-alignment image (reference sign y in FIG. 15) are collated by template matching, and the matching degree between the collated images is the string detection degree d + calculated as.

[0056] In the second string inspection method, after generating the pseudo-alignment image in S32, the display string "area classification" in the screen component object is extracted (reference sign b in FIG. 15), and this display string "area classification" and the string shown in the pseudo-alignment image are collated by OCV, and it is verified whether the display string "area classification" in the above screen component object is drawn in the pseudo-alignment image. The success degree of this collation is the string detection degree d + calculated as.

[0057] In the third string inspection method, after generating the pseudo-alignment image in S32, the display string "area classification" in the screen component object is extracted (reference sign c in FIG. 15), and the string shown in the pseudo-alignment image is read by OCR. Then, this read string and the display string "area classification" in the above screen component object are collated, and the similarity between the two is the string detection degree d​+ It is calculated as

[0058] Next, another example of the details of the generation of the above alignment criterion d upper will be described. This example is an improved version of the example described with reference to FIGS. 14 and 15. FIG. 16 is a flowchart showing an example of a processing operation related to the calculation of the string detection degree reduction rate by the screen data processing apparatus. Since the above pseudo-alignment image is an image with a relatively simple background, the string detection degree d + tends to be calculated as a relatively high value according to this image.

[0059] FIG. 17 is a diagram showing an example of the string detection degree calculated for the pseudo-alignment image. In FIG. 17, the alignment criterion d lower calculated for the pseudo-mismatch image, the actual string detection degree d calculated for the screen component, and the string detection degree d + calculated for the pseudo-alignment image are shown. However, as described above, when the string detection degree d + is calculated as a relatively high value, the average of the alignment criterion d lower and the string detection degree d + calculated for the pseudo-alignment image also becomes a high value, and the string detection degree d calculated for the matching screen component falls below the average, and a case where it is determined that "no string drawing" is likely to occur.

[0060] Therefore, in the improved example, prior to the inspection of the consistency of the screen data, a mechanism ( = alignment image collection function, described later) for collecting alignment images that satisfy all of the following alignment image conditions (a), (b), and (c) without requiring work only for this purpose is prepared (S41).

[0061] <Alignment Image Conditions> (a) The displayed string is known. (b) It is known that the displayed string is drawn in the screen image. (c) In the screen image, the drawing areas of the screen components are known.

[0062] For one or more aligned images collected as described above, for each aligned image, first, the character string detection degree d for the pseudo-aligned image is calculated by the detection degree reduction rate calculation unit 17 + is calculated, and second, the character string detection degree d for the aligned image is derived.

[0063] The detection degree reduction rate calculation unit 17 calculates the character string detection degree reduction rate r as d / d + and holds the calculation result in the detection degree reduction rate holding unit 18 (S42). When there are a plurality of aligned images, the worst value, average value, or median value among the character string detection degrees calculated for each aligned image may be used in subsequent processing.

[0064] When inspecting the consistency of the screen data, for each screen component to be inspected, the value in which the character string detection degree reduction rate r calculated above is reflected in the character string detection degree d + calculated using the pseudo-aligned image is used as the reference d upper at the time of alignment. That is, d upper = r × d + is used. The character string detection degree itself varies greatly depending on cases such as character strings, but the character string detection degree reduction rate is less affected.

[0065] FIG. 18 is a diagram showing an example of the reference at the time of alignment calculated according to the character string detection degree reduction rate. In the example shown in FIG. 18, the character string detection degree d is calculated for each of three types of screen components managed by the information of the screen component object, here the screen component A, the screen component B, and the screen component C. Further, the character string detection degree d + corresponding to the screen component A and the pseudo-aligned image and the reference d lower at the time of non-alignment are calculated, and the character string detection degree d + corresponding to the screen component B and the pseudo-aligned image and the reference d loweris calculated, and the string detection degree d corresponding to the screen component C and the pseudo-matching image + and the standard d at the time of mismatch lower are calculated. The string detection degree d and the standard d at the time of mismatch shown in this Fig. 18 lower are the same size as the string detection degree d and the standard d at the time of mismatch shown in Fig. 11 lower is the same.

[0066] Furthermore, in the example shown in Fig. 18, the string detection degree d calculated according to the screen component A + and the standard d at the time of match according to the screen component A based on the calculated string detection degree reduction rate r upper , the string detection degree d calculated according to the screen component B + and the standard d at the time of match according to the screen component B based on the calculated string detection degree reduction rate r upper , and the string detection degree d calculated according to the screen component C + and the standard d at the time of match according to the screen component C based on the calculated string detection degree reduction rate r upper are shown as examples. The size of the string detection degree d shown in Fig. 18 + is the same size as the standard d at the time of match shown in Fig. 11 upper is the same.

[0067] The string detection degree d calculated for the screen component A is between the standard d at the time of mismatch calculated for the screen component A lower and the standard d at the time of match upper , and is closer to the standard d at the time of match lower compared with the standard d at the time of mismatch upper , that is, higher than the average of the standard d at the time of mismatch lower and the standard d at the time of match upper . Therefore, for this screen component A, the determination result of whether to draw the display string is "display string drawing exists". As a result, for the example shown in Fig. 11 where the determination result of whether to draw the display string for the screen component A was "no display string drawing", this example and the string detection degree d and the standard d at the time of mismatch lowerUnder the condition that the sizes are the same, the criterion d at the time of alignment upper In the example shown in FIG. 18 newly calculated, it is shown that the determination result of whether or not to draw the display string for the screen component A has been changed to "there is display string drawing".

[0068] Also, the string detection degree d calculated for the screen component B is the criterion d at the time of alignment calculated for the screen component B upper Since it is higher, for this screen component B, the determination result of whether or not to draw the display string is "there is display string drawing". The string detection degree d calculated for the screen component C is the criterion d at the time of misalignment calculated for the screen component C lower and the criterion d at the time of alignment upper Between, and the criterion d at the time of misalignment lower Compared with the criterion d at the time of alignment upper Is close to, that is, the criterion d at the time of misalignment lower And the criterion d at the time of alignment upper Since it is higher than the average of, for this screen component C, the determination result of whether or not to draw the display string is "there is display string drawing".

[0069] Next, the details of the collection of the above alignment images will be described. FIG. 19 is a diagram showing an example of the collection of alignment images. As the alignment image collection function, for example, past screen data obtained by a screen data acquisition program, that is, information including a screen image and information on screen component objects is used.

[0070] The user or the counselor matching device inputs (in (1) of FIG. 19) the extraction conditions of the screen data, that is, the conditions of the desired screen data among the screen data held in the screen data holding unit 15. According to this condition, the screen data extraction unit 12 extracts screen data from the set of screen data (d1 in FIG. 19) held in the screen data holding unit 15, and the screen image in this screen data is presented to the user ((2) in FIG. 19) ((3) in FIG. 19). This enables the user to refer to the screens that have been displayed in the past, without the user having to search for communication tools, materials, etc. on their own.

[0071] When referring to the screens that have been displayed in the past, in many cases, the user wants to use a specific display string drawn on the screen image. Here, through an input operation by the user, in the screen image of the past screen data displayed on the screen data display unit 16, a desired point or area where the above-mentioned string to be used is drawn is specified ((4) in FIG. 19).

[0072] The screen data display unit 16 selects (identifies) the screen component corresponding to the specified point or area from the information of the screen component objects held in the screen data holding unit 15, and highlights or emphasizes the drawing area of this screen component with a highlight or a rectangular frame so that the user can visually recognize it ((5) in FIG. 19).

[0073] The copy operation detection unit 16-1 of the screen data display unit 16 detects an input operation related to copying the above-mentioned highlighted string by the user, and the display string copying unit 16-2 copies the display string of the above-mentioned highlighted screen component ((6) in FIG. 19). The display string copying unit 16-2 temporarily stores the copied display string of the screen component in a clipboard (clipboard) (d2 in FIG. 19) ((7) in FIG. 19).

[0074] The matching image recording unit 16-3 determines that the image within the drawing area of the screen component for which this storage has been made satisfies the conditions of a matching image, extracts the corresponding image within the drawing area as a matching image, and stores it in the matching image holding unit 19 ((8) in FIG. 19).

[0075] Here, in order to collect many cases in a short period, for screen components other than the screen components to be copied included in the same screen data, they may also be added as matching images. Further, in order to further enhance the accuracy of determining whether or not the conditions for the matching images are satisfied, whether or not the above-mentioned copied screen components have been pasted may be added to the above-mentioned determination conditions.

[0076] Then, a pseudo-matching image corresponding to the display string of the screen component selected (specified) from the information of the screen component object is generated by the pseudo-matching image generation unit 13-5 (Figure 19(9)), and the string detection degree d corresponding to this pseudo-matching image is calculated by the detection degree reduction rate calculation unit 17 + is calculated, and the string detection degree d corresponding to the matching image held in the matching image holding unit 19 as described above is calculated (Figure 19(10)).

[0077] And the calculated string detection degree d + and the string detection degree d, a string detection degree reduction rate r is calculated (Figure 19(11)), and according to this string detection degree reduction rate r, the matching standard d at the time of matching described above upper is calculated, and a determination as to whether or not to draw the display string is made.

[0078] Figure 20 is a block diagram showing an example of the hardware configuration of a screen data processing device according to an embodiment of the present invention. In the example shown in the figure, the screen data processing device 100 is composed of, for example, a server computer or a personal computer, and has a hardware processor 111 such as a CPU (Central Processing Unit). Then, a program memory 111B, a data memory 112, an input / output interface 113, and a communication interface 114 are connected to this hardware processor 111 via a bus 120.

[0079] The communication interface 114 includes, for example, one or more wireless communication interface units, enabling the transmission and reception of information with the communication network NW. As the wireless interface, an interface adopting a low-power wireless data communication standard such as a wireless LAN (Local Area Network) can be used.

[0080] Connected to the input / output interface 113 are an input device 130 for an operator and an output device 140 attached to the screen data processing device 100. The input / output interface 113 captures operation data input by the operator through the input device 130 such as a keyboard, touch panel, touchpad, mouse, etc., and performs processing to output and display the output data to the output device 140 including a display device using liquid crystal or organic EL (organic electro-luminescence), etc. Note that the input device 130 and the output device 140 may use devices built into the screen data processing device 100, or the input devices and output devices of other information terminals capable of communicating with the screen data processing device 100 via the communication network NW may also be used.

[0081] The program memory 111B is used in combination as a non-temporary tangible storage medium, for example, a non-volatile memory such as an HDD (Hard Disk Drive) or SSD (Solid State Drive) that can be written and read at any time, and a non-volatile memory such as a ROM (Read Only Memory), and can store programs necessary for executing various processes according to an embodiment.

[0082] The data memory 112 is a tangible storage medium, for example, a combination of the above-mentioned non-volatile memory and a volatile memory such as RAM (Random Access Memory), and can be used to store various data acquired and created during the process of performing various processes.

[0083] The screen data processing apparatus 100 according to an embodiment of the present invention can be configured as a data processing apparatus having, as software-based processing functional units, an input unit 11, a screen data extraction unit 12, a screen data integrity inspection unit 13, a screen data exclusion or selection unit 14, a screen data display unit 16, and a detection degree reduction rate calculation unit 17 shown in FIG. 1.

[0084] Also, the working memory used for various processes in the screen data processing apparatus 100, and the screen data holding unit 15, the detection degree reduction rate holding unit 18, the integrated image holding unit 19, and the screen data integrity degree holding unit 20 shown in FIG. 1, and the various holding units in the screen data integrity inspection unit 13 can be configured using the data memory 112 shown in FIG. 20.

[0085] The processing functional units in each of the above-mentioned units such as the screen data extraction unit 12, the screen data integrity inspection unit 13, the screen data exclusion or selection unit 14, the screen data display unit 16, and the detection degree reduction rate calculation unit 17 shown in FIG. 1 can all be realized by causing the hardware processor 111 to read and execute the program stored in the program memory 111B. Note that some or all of these processing functional units may be realized in other various forms including integrated circuits such as an application specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0086] In addition, the methods described in each embodiment can be implemented as a program (software means) to be executed by a computer, and stored, for example, in a recording medium such as a magnetic disk (e.g., a floppy disk, hard disk, etc.), an optical disc (e.g., CD-ROM, DVD, MO, etc.), or a semiconductor memory (e.g., ROM, RAM, flash memory, etc.). It can also be transmitted and distributed via a communication medium. Note that the program stored in the medium includes not only software means to be executed by the computer (including not only the execution program but also tables and data structures), but also a setting program for configuring software means in the computer. The computer that realizes this device reads the program recorded in the recording medium, and in some cases, constructs software means using the setting program, and executes the above-described processing by controlling the operation with this software means. Note that the recording medium referred to in this specification includes not only a medium for distribution, but also storage media such as magnetic disks and semiconductor memories provided inside the computer or in devices connected via a network.

[0087] Note that the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the gist thereof at the implementation stage. Also, the embodiments may be implemented in appropriate combinations, and in such cases, the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by combinations selected from a plurality of disclosed constituent elements. For example, even if some constituent elements are deleted from all the constituent elements shown in the embodiments, and the problem can be solved and the effects can be obtained, the configuration from which these constituent elements are deleted can be extracted as an invention.

Description of Reference Numerals

[0088] 100... Screen data processing device 11... Input unit 12... Screen data extraction unit 13... Screen data integrity inspection unit 13-1... Inspection target area selection unit 13-2...Pseudo mismatch image generation unit 13-3…Inconsistency guideline calculation section 13-4…Inconsistency reference holding section 13-5…Pseudo matching image generation unit 13-6…Consistency guideline calculation section 13-7…Alignment guideline holding section 13-8…Display character string detection degree calculation section 13-9…Display character string presence / absence determination section 13-10...Screen data consistency calculation section 14…Screen data exclusion or selection section 15...Screen data storage section 16...Screen data display section 16-1…Copy operation detection unit 16-2…Display string duplication section 16-3…Integrated image recording section 17…Detection rate reduction calculation section 18…Detection rate reduction rate storage section 19...Integrated image storage unit 20...Screen data consistency holding unit

Claims

1. A detection degree calculation unit that manages information on a character string that is a component of screen data, calculates a character string detection degree indicating how much the character string managed by the screen component information is drawn in the area, based on the screen component information obtained together with an image from a display screen and a partial area of the image displayed on the display screen; A determination unit that determines whether or not the character string managed by the screen component information is drawn in the area, based on a comparison between the character string detection degree calculated by the detection degree calculation unit and a threshold value; A screen data processing apparatus comprising the above.

2. The screen component information is information that manages the information on the character string for each of a plurality of components of the screen data, The determination unit, determines for each of the plurality of components whether or not the character string managed by the screen component information is drawn in a partial area of the image displayed on the display screen, and further comprises a consistency calculation unit that calculates a screen data consistency indicating a ratio of components determined to have the character string managed by the screen component information drawn in a partial area of the image displayed on the display screen, among the plurality of components, according to the determination results for each of the plurality of components by the determination unit; The screen data processing apparatus according to Claim 1.

3. A first generation unit that generates a pseudo-inconsistent image having the same size as the drawing area of the screen component managed by the screen component information and corresponding to an area that does not match the screen component; A second generation unit that generates a pseudo-consistent image having the same size as the drawing area of the screen component managed by the screen component information and corresponding to an area that matches the screen component; A first calculation unit that calculates a character string detection degree indicating how much the character string managed by the screen component information is drawn in the pseudo-inconsistent image generated by the first generation unit; A second calculation unit that calculates a character string detection degree indicating how much the character string managed by the screen component information is drawn in the pseudo-consistent image generated by the second generation unit, and further comprises: The determination unit, determines whether or not the character string managed by the screen component information is drawn in the area, based on the relationship between the character string detection degree calculated by the first calculation unit, the character string detection degree calculated by the second calculation unit, and the character string detection degree calculated by the detection degree calculation unit. The screen data processing apparatus according to claim 1.

4. A third calculation unit that calculates a string detection degree indicating to what extent a character string managed by the screen component information is drawn in a matching image that matches the screen component managed by the screen component information; A decrease rate calculation unit that calculates a decrease rate from the string detection degree calculated by the second calculation unit to the string detection degree calculated by the third calculation unit based on the relationship between the string detection degree calculated by the second calculation unit and the string detection degree calculated by the third calculation unit; and The determination unit Based on the relationship between the string detection degree calculated by the second calculation unit, the value changed by the decrease rate calculated by the decrease rate calculation unit, the string detection degree calculated by the first calculation unit, and the string detection degree calculated by the detection degree calculation unit, it is determined whether or not the character string managed by the screen component information is drawn in the area. The screen data processing apparatus according to claim 3.

5. A screen data holding unit that holds past screen data; An alignment image extraction unit that extracts, as the alignment image, an image of an area where a desired screen component in the held screen data is drawn; and The screen data processing apparatus according to claim 4.

6. A method performed by a screen data processing apparatus, Based on the screen component information in which information on a character string that is a component of the screen data is managed and acquired together with an image from a display screen, and a partial area of the image displayed on the display screen, the detection degree calculation unit of the screen data processing apparatus calculates a string detection degree indicating to what extent the character string managed by the screen component information is drawn in the area; Based on a comparison between the string detection degree calculated by the detection degree calculation unit and a threshold value, the determination unit of the screen data processing apparatus determines whether or not the character string managed by the screen component information is drawn in the area; A screen data processing method comprising:

7. The screen component information is information in which the information on the character string is managed for each of a plurality of components of the screen data, The determination unit Determines for each of the plurality of components whether or not a character string managed by the screen component information is drawn in a partial area of the image displayed on the display screen. The screen data processing apparatus further includes a consistency calculation unit that calculates a screen data consistency indicating a ratio of components determined to have a character string managed by the screen component information drawn in a partial area of an image displayed on the display screen among the plurality of components, according to the determination results for each of the plurality of components by the determination unit. The screen data processing method according to claim 6.

8. A screen data processing program that causes a processor to function as each part of the screen data processing apparatus according to any one of claims 1 to 5.

Images