Image-based program state analysis system and method

Abstract

An image-based program state analysis method comprising the steps of: for a given image generated by a program, identifying at least a first location within the image occupied by a respective user interface element; identifying a predetermined point on a corresponding user interface element in the image; identifying information communicated by the corresponding user interface element with reference to the predetermined point; and outputting the information for subsequent analysis.

Description

Technical Field

[0001] The invention relates to an image-based program state analysis system and method. Background Art

[0002] Game developers, publishers, or platform managers of video game platforms on which games are played often wish to obtain telemetry data about gameplay to identify potential bugs, game issues, player behavior and interactions within the game, and any unexpected or unknown aspects of the game that would only become apparent if a significant number of people played the game. To this end, many modern games provide such telemetry data to at least one of the developers, publishers, and platform managers, but unfortunately, this is not the case for all games, so it is desirable to have alternative methods when the party that desires telemetry data or sufficient telemetry data is not available.

[0003] The present invention aims to solve or alleviate this problem. Summary of the invention

[0004] Various aspects and features of the present invention are defined in the accompanying claims and in the text of the accompanying description.

[0005] In a first aspect, according to claim 1, there is provided an image-based program state analysis method.

[0006] In another aspect, an image-based program state analysis system is provided according to claim 16. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] A more complete appreciation of the present disclosure and many of the attendant advantages thereof will be more readily obtained and become better understood by reference to the following detailed description when considered in conjunction with the accompanying drawings, in which:

[0008] Figure 1 is a schematic diagram of an entertainment device according to an embodiment of the present specification.

[0009] Figure 2 is a schematic diagram of an image generated by a program according to an embodiment of the present specification, including a plurality of corresponding user interface elements arranged in what is called a heads up display (HUD).

[0010] Figure 3 is a flowchart of an image-based program state analysis method according to an embodiment of the present specification. DETAILED DESCRIPTION

[0011] A system and method for analyzing program state based on an image are disclosed. In the following description, many specific details will be presented to provide a thorough understanding of embodiments of the present invention. However, it should be understood by those skilled in the art that these specific details are not necessary for implementing the present invention. On the contrary, for the sake of clarity, specific details known to those skilled in the art are appropriately omitted.

[0012] Referring now to the drawings, in which like reference numerals designate identical or corresponding parts throughout the several views, Figure 1 An example of an entertainment system 10 is shown, in this case a (PS5) computer or console.

[0013] The image-based program state analysis system may be included in such an entertainment system 10, or equivalently, the image-based program state analysis method may be executed by such an entertainment system 10 (eg, under suitable software instructions).

[0014] The entertainment system 10 includes a central processor 20. The processor can be a single-core or multi-core processor, for example, the PS5 includes an octa-core processor. The entertainment system also includes a graphics processing unit or GPU 30. The GPU can be physically separated from the CPU, or integrated with the CPU as a system on a chip (SoC) such as in the PS5.

[0015] The entertainment device also includes RAM 40, and each CPU and GPU can have independent RAM, or share RAM as in PS5. Each RAM can be physically independent, or integrated as part of the SoC as in PS5. Disk 50 provides further storage, and the disk can be an external or internal hard disk, an external solid-state drive, or an internal solid-state drive as in PS5.

[0016] The entertainment device can be connected to the computer through one or more data ports 60 (such as USB ports, port, port, Port or similar port, as the case may be) to transmit or receive data. Optionally, data can also be received via the optical drive 70.

[0017] Audio / video output from the entertainment device is typically provided through one or more A / V ports 90 or through one or more wired or wireless data ports 60 .

[0018] One example of a device for displaying images output by an entertainment system is a head mounted display (HMD) 120 worn by the user 1 , such as a PlayStation VR 2 (PSVR 2).

[0019] When components are not integrated, the components may be connected in a suitable manner through dedicated data links or via bus 100 .

[0020] Interaction with the system is typically provided using one or more handheld controllers (130, 130A), such as the Controller (130), and / or one or more VR controllers (130A-L, 130A-R), for example in the case of an HMD.

[0021] Heads-up display

[0022] refer to Figure 2 , a typical video game includes visual feedback overlaid on the game action in the form of a heads-up display or "HUD". Visual feedback is usually intended to provide the user with immediate information about various aspects of the game, such as the player's health or stamina in the game (e.g. Figure 2 HUD area 210 in the image), inventory items they have equipped or can use (e.g. Figure 2 220) and any other in-game information, such as opponent health values ​​(e.g. Figure 2 230 in the HUD area).

[0023] In the first embodiment of this specification, these HUD areas occupy fixed positions on the screen, although optionally certain HUD areas are otherwise conditional / temporary (such as HUD area 230, which only appears when there is an opponent with an associated health bar).

[0024] Since different games have different HUDs, in the embodiments of this specification, a game-specific HUD configuration file is associated with the corresponding game, and the corresponding game is used to define which HUD elements in the game are associated with the purpose of image-based program state analysis. The configuration file can be generated by the game developer, publisher, platform manager or other third party.

[0025] The configuration file may define the area of ​​the screen occupied by the HUD element and, optionally, a root position or anchor point for the starting point of the HUD element. Typically, the starting point corresponds to 0% on a bar or other representation of a continuous variable in the HUD, or to a fixed point from which image data in the area is extracted for comparison with the appearance of reference image data, or at which an optical character recognition process is started. The area and / or position may be defined in a resolution-independent manner, for example as a percentage of the distance from a reference corner of the screen in each direction. Alternatively or additionally, different areas and / or positions may be provided for different resolutions if at least a portion of the HUD is laid out differently for different corresponding resolutions.

[0026] Optionally, the profile can also define the type of HUD element to inform the appropriate data extraction type; for example, it can define a bar element, an image type element, a numeric character element, and an alphanumeric character element. The profile can also define whether a hard element is conditional / temporary or permanent.

[0027] Bar elements

[0028] For bar elements, based on the defined area and / or defined appearance of the bar within the area and the starting point of the bar element, the analysis system can evaluate the current image to determine how much of the maximum bar is currently displayed in the game, thereby determining the percentage of the screen currently represented by the bar or an equivalent value. Figure 2 , three bars are displayed in the HUD area 210. In this case, the health bar reads as 98% full; the mana bar (mana is usually magic recharge) reads as 61% full, and the stamina bar reads as 98% full.

[0029] It's worth noting that the health bar and stamina bar both read as 98% full, but they are physically different lengths because the maximum lengths of the corresponding bars are different.

[0030] In the case of a game where the default length of such a bar can be extended (e.g. to 120% of the maximum health), then in principle the area including the health bar can be defined to accommodate the maximum extended health bar, and the analysis system reads the percentage up to the maximum extension percentage. Alternatively or additionally, if the area including the health bar goes up to 100%, but in the game the health bar can be extended to 120%, then the analysis system can detect when the health bar is extended beyond the default area and extrapolate the value accordingly.

[0031] It should be appreciated that in some games the length of the bar remains constant but the color components within the bar change. For example, at 100% the bar is completely green, while at 75% health three quarters of the bar is green and the last quarter of the bar is red. This can be considered essentially the same scenario, with the length of the green bar being the object of evaluation. In both cases the color of the remaining area (whether red or part of the virtual environment) is less relevant than the color of the bar. Finally, in cases where the bar also changes color based on health as it grows or shrinks, typically the game ensures that the bar is visible and therefore places it within a high contrast boundary (which in turn can be used to define the area for analysis), or is distinguished from the background environment by brightness, for example.

[0032] Optionally, the image may be analyzed within the luminance channel or other grayscale representation if appropriate for the game, which may also be an option indicated in the configuration file.

[0033] In any case, the image analysis system can perform image analysis by detecting the brightness and / or color of the pixel at the starting point of the bar, and measuring how far along the bar similar pixels extend to measure the current range of the bar, and thereby calculating the current value represented by the bar in conjunction with the maximum range of the bar. Alternatively or additionally, the configuration file can specify the brightness and / or color of the pixels expected in the bar, thereby enabling confirmation of the starting point, or avoiding the need to identify the expected color of the rest of the bar.

[0034] In a bar variation, a discrete element (e.g., a heart icon, etc.) may be displayed that indicates, for example, the number of hits a character can take before dying. In this case, the profile may indicate the presence of a discrete bar and the relative emptiness of the discrete element. The image analysis system may then detect the presence or absence of the icon in the empty space to determine the effective range of the bar in a manner similar to that described above.

[0035] Thus, more generally, the image analysis system performs image analysis to detect how much of the largest bar at a defined position is currently depicted on the screen, and thereby estimate the value of the in-game attribute represented by that bar.

[0036] Image type element

[0037] Alternatively or in addition to the other elements described herein, the HUD may include graphical elements. Typically, but not exclusively, these graphical elements relate to equipped inventory items and / or equipped spells or other abilities.

[0038] It should be noted that the currently displayed graphical elements may represent only a small fraction of the number of graphical elements that may be displayed. For example, a player may have dozens or hundreds of items that can be equipped, but as a non-limiting example, the HUD only displays four equipment slots.

[0039] So in this case, the image analysis system can extract or isolate the image of the item from the corresponding equipment slot and compare it to the possible items that can be equipped to detect what the user currently has equipped. This can be performed as template matching, for example.

[0040] To make this process efficient, the image analysis system may subsample or otherwise reduce the image extracted from the equivalent slot to a small image, and / or generate a perceptual hash or other direct or indirect representation of the image for efficient comparison with an equivalent representation of a reference item.

[0041] Once a match is found (or a close enough match within a tolerance threshold), the image analysis system obtains the item definition associated with the matching reference item.

[0042] Representations of reference items may be associated with the profiles. Alternatively or additionally, they may be stored centrally, for example, as a server of a platform manager, developer and / or publisher, and representations of images are uploaded to the server as queries, and item definitions associated with matching reference items are returned as responses. Thus, in this case, the images are identified at the central server.

[0043] It will be appreciated that some of the graphical elements may represent one of a large number of alternatives (such as in the case of equipped items), so it is more efficient to query a central repository of reference items, while other of the graphical elements may represent one of a small number of alternatives (e.g., in the case of a pictorial representation of a user's health in a game, or a representation of progress through a goal, such as in the form of accumulated fragments of keys or other items), and so optionally, representations of some reference items may be stored locally, while representations of other reference items may be stored centrally. In this case, the configuration file will typically indicate where the representations of the reference items are stored.

[0044] In any case, optionally, the image analysis system periodically checks the or each equipment slot and attempts to identify an image in the equipment slot only if the image has changed or has changed by more than a threshold amount and / or for a threshold period of time.

[0045] In the case where inventory items are displayed on a transparent background (so that the image within the equipment slot is constantly changing due to changes in the background), assuming that the equipped item does not change at the same frequency as changes in the background, an invariant component of the equipment slot image (i.e., corresponding to the actual item itself) can be identified, and changes relative to that invariant component can be detected (and optionally, only a representation of the invariant component can be used for comparison with a representation of a reference item).

[0046] It should be understood that references to "equipment slots" are merely exemplary and merely illustrate the concept of defined areas where graphical type elements may be found within the HUD.

[0047] Thus, more generally, the image analysis system performs image analysis to identify image elements within a defined area of ​​the HUD by comparing representations of image type elements to corresponding representations of multiple reference image elements and obtaining an item definition associated with the best matching reference item (where best may further require an exact match, a match within a tolerance threshold, or simply a best match).

[0048] Numeric character elements

[0049] Alternatively or in addition to other elements described herein, the HUD may include numeric character elements (e.g., characters 0-9, and optionally some common numeric separators, such as a decimal point (.), a comma (,), a division sign ( / ), or a ratio (:), and similarly optionally some common measurements, such as seconds (s), meters (m), percentages (%), revolutions per minute (rpm), etc.

[0050] Often (but not always) for numeric character elements their positions are fixed, either the hundreds, tens and ones (and any other numeric positions) are assigned fixed positions, or the numbers are counted upwards from the starting position.

[0051] Thus, in either case, the profile may identify the starting points of the numeric character elements, and optionally each element may be identified individually, or the total may be identified (although in practice the identification process may be similar in both cases).

[0052] Each digital character element may be identified using optical character recognition for the individual or collective area occupied by each digital character element. Typically, the profile will identify what the digital character element represents (e.g., time or distance), so there is no need to identify the measurement associated with the HUD itself. However, this is also an option.

[0053] The configuration file may include data related to a particular font for optical character recognition (e.g., providing template numeric character elements, or a configured recognition engine (e.g., previously trained weights of a machine learning algorithm or a hidden Markov model). However, typically for numeric character elements, specific modifications to the font may not be necessary because the number of characters is small and they are clearly distinguishable from each other.

[0054] Optionally, the configuration file may indicate where HUD elements are cross-referenced with each other; for example, if the player's health is represented by a bar element and a numeric character element, one of the elements may be omitted from detection as redundant, or both may be detected and used to cross-validate each other. More generally, a HUD element may simply indicate that a numerical value may have changed, and may indicate the direction of the change. For example, if a HUD element indicates that damage has been taken, then the numerical representation of health will likely go down rather than up, and this may be used to validate OCR results.

[0055] Thus, more generally, the image analysis system performs image analysis using optical character recognition to identify one or more digital character elements within a defined area of ​​the HUD (or equivalently starting from a defined starting point on the HUD).

[0056] While as described above digits may have fixed positions, this is not always the case, and so digits (including the case of only digits) may alternatively be treated similarly to more general alphanumeric character elements, as described below.

[0057] Alphanumeric character elements

[0058] Alternatively or in addition to other elements described herein, the HUD may include alphanumeric character elements, such as letters and optionally numbers, and also optionally certain punctuation marks (eg, periods, commas, question marks, exclamation points, etc.).

[0059] The process of identifying these is generally similar to the process of identifying digital character elements described elsewhere herein, and may optionally also include correction data for specific fonts as described elsewhere herein.

[0060] However, typically for alphanumeric character elements, their positioning depends on the so-called text alignment; this may be aligned to the left, right (particularly for certain languages ​​that are read from right to left), top (particularly for certain languages ​​that are read vertically) or centered in a predetermined area, which may be particularly common for annotations to gaming items, so that they are aligned with the item's associated description or an associated bar element, etc.

[0061] Alphanumeric character elements are more likely to be part of variable-length text than numeric character elements, so the overall area where these characters can be found may be larger but not completely filled. In this sense, a strategy similar to strip elements (or discrete strip elements) can be considered based on the progression from the starting point.

[0062] Therefore, for left-aligned, right-aligned, top-aligned or bottom-aligned text, the starting point of OCR can be located at the associated left, right, top or bottom of the predetermined area, and then proceed according to the expected reading direction. At the same time, for center-aligned text, the starting point can be located at the center within the predetermined area (or the first line of the expected predetermined area), and OCR can be used to determine the scope of the text at least on the left (for text read from left to right) or on the right (for text read from right to left), etc., and once the starting point has been identified, the text can be read with OCR. As described below, if the text that has been read in the process of determining the starting point appears to be valid, it does not need to be re-read.

[0063] Alternatively, OCR can start from the relevant edge of the overall region appropriate to the current language, and continue searching until the first character or word is recognized, and then continue reading.

[0064] For either case of center-aligned text, the assumed symmetry of text length, at least within a tolerance threshold, can optionally help eliminate any false positives when searching for alphanumeric characters.

[0065] OCR typically recognizes individual characters, but can then refine that recognition, or re-recognize at a global level, based on whether the combined characters correspond to dictionary words or (in text that may contain more than just dictionary words, such as in a game) correspond to text that follows expected pronunciation constructions and / or is based on common double or triple letters. Optionally, in a similar manner to the image type elements described elsewhere herein, the profile can include a dictionary or vocabulary of words expected to be found in part or all of the content of the game.

[0066] In some cases, only keywords within the text may be relevant, such as, for example, a character's name. What the character says (if anything) may optionally be considered unimportant. Alternatively or additionally, other key items (such as specific quest items) may also be considered relevant, so when a character mentions these key items, multiple pieces of relevant information may appear in a single block of text. Related content is described elsewhere herein.

[0067] Thus, more generally, when an image analysis system performs image analysis to identify one or more alphanumeric character elements in a defined region of a HUD, a search strategy responsive to expected text alignment within that region is optionally employed.

[0068] Other Elements

[0069] The examples of bar elements, image type elements, numeric character elements, and alphanumeric character elements are non-limiting examples, and any type of HUD element may be considered. For example, the color of a HUD element may change to indicate an event such as a player's character being hit. In this case, one or more sample pixels of the relevant HUD element may be tracked to detect the relevant color change, and the profile may identify the pixel or associated area from which the sampling was performed. Other HUD elements will be apparent to the skilled person.

[0070] Conditional / temporary elements.

[0071] As described elsewhere in this article, HUD elements are generally static and permanent (even if the content changes). However, some HUD elements are temporary and / or conditional. Figure 2 A common example is Figure 2The health bar for a boss enemy only appears when the user is engaged in combat with the boss, and also provides alphanumeric text identifying the boss' name, so in this case it is also an example of a temporary and / or conditional HUD element.

[0072] Other examples include additional HUD elements that are added when a user gains or equips a new skill or gear, and which can then be permanent or disappear when the user equips the skill or gear (or in some cases where the game removes the skill or gear).

[0073] Another example is when a game contains multiple playable characters, those characters may have different fixed elements associated with them (e.g., for the reasons given above relating to different skills or equipment), and so the HUD would change when the user selects between them.

[0074] In these cases, the image analysis system may periodically check for characteristic aspects of the HUD element at the expected location (e.g., looking for a green or related color at the start of a boss's health bar, or if the health bar is variable and centered, looking for a green color at the center point of the boss's health bar (assuming the boss starts at full health)). Optionally, to avoid false positives, as with image type elements, image data for a sub-region of the HUD may be compared to a reference image of the HUD to identify the presence or absence of the HUD element. Thus, for example, when a green health bar with white borders above and below is present, a sub-region of the image that is expected to contain that image style when a hunt element is presented may be compared to a reference image of that image style to detect whether the hunt element is present. This is likely to avoid false positives caused by, for example, a player running on grass, which may be green but is unlikely to generate the specific green color with white borders above and below that is associated with the HUD element itself. In the case where the HUD element is simply numeric or alphanumeric text, OCR may be used to detect only whether any text is currently presented within the expected area (e.g., a sub-region that is one or two characters wide).

[0075] Once HUD elements have been detected, they can be evaluated using the relevant techniques described elsewhere in this document.

[0076] If a fixed element appears and then disappears, this may be indicated by the image analysis system's failure to determine a value (or expected / valid value.) For such temporary staff conditional HUD elements, this may be interpreted as the disappearance of the element, in which case the system may resume detecting the reappearance of that HUD element.

[0077] In a sense, all HUD elements in a game running (or a game environment in which the HUD is displayed running) are temporary or conditional, so the disappearance of all HUD elements at substantially the same time can indicate that the game is in a switching state (e.g., switching to a menu or other user interface) or the game is closed.

[0078] Thus, more generally, for temporary / conditional HUD elements, the image analysis system performs image analysis to detect the presence of the HUD element prior to performing one of the other recognition strategies described elsewhere herein, and for these elements, individually interprets a failure to recognize an expected / valid value as indicating that the HUD element has disappeared again. Also, individually, optionally, if all or a threshold number of HUD elements disappear substantially simultaneously, this may indicate a change in the mode of play or use of the game.

[0079] Analysis frequency

[0080] Different elements of the HUD can be evaluated at different periods. For example, changes in health can indicate an emergency, and so it is best to track health every frame or every few frames to monitor its state at a high frequency. Since health is usually depicted by a health bar, which is easy to detect and evaluate, this has a relatively low computational overhead.

[0081] In contrast, the relatively more computationally expensive process of identifying image-type elements can be performed at a slower period (e.g., once every 1, 10, or 60 seconds) and / or, as described elsewhere in this document, conditionally performed based on whether a change or sufficient change is detected in an element (typically an identified element).

[0082] Alphanumeric text is also relatively computationally expensive and may employ similar strategies as image-type elements, such as detecting every one or several seconds (e.g., a period slightly shorter than the typical reading time of a sentence of expected length within a predetermined region of alphanumeric text), and / or conditionally indicating when text appears or when text changes based on whether a change or sufficient change is detected within a predetermined region, as described elsewhere in this document.

[0083] Numeric text may be slightly easier to detect than alphanumeric text, and generally follows more predictable rules. For example, when the numbers indicate distance traveled, it may not be necessary to re-detect the high-order digits periodically until the amount indicated indicates that these numbers will change. So, for example, if the current distance of the race is 1050 meters, then the beginning "105" may not need to be re-detected until the last character is detected as a 9 and then a 0. Similarly, the leading "1" may not need to be re-detected until the other three characters are 9 9 9 and then all 0. Alternatively, these numbers may be re-evaluated at a relatively low rate, either way just to correct any possible errors or identification discrepancies.

[0084] The numbers may also be evaluated at different periods depending on their characteristics or significance. For example, if the numbers are a countdown timer, the last few seconds may be important, so the frequency of evaluating the numbers may be increased up to and including the frame rate. Similarly, integers such as 10, 100, or 1000 may be more significant to the user than other numbers, so the period of identification may be increased as the number character set approaches such integers or other significant numbers identified in the profile.

[0085] Finally, HUD elements can be re-evaluated in response to the detection of events. For example, if a HUD element disappears or invalid output is generated for it, the HUD element can be evaluated to determine, for example, whether the game changes state with the associated large-scale batch change or deletion of the HUD element. Similarly, when there is cross-validation between HUD elements, for example, a change in one element (such as a bar element) evaluated at a high frequency can trigger a re-evaluation of related elements (such as a numeric character element), and other elements can be evaluated at a lower frequency.

[0086] Thus, more generally, HUD elements may be identified with a common periodicity, or with a periodicity that depends on one or more of: likelihood of change, computational cost of performing identification, detection of events that may cause change, and rules associated with values ​​(e.g., based on counts or significance).

[0087] Dynamic HUD elements

[0088] The above description assumes that HUD elements are static, or at least the area they occupy is static, even if the HUD elements themselves move position or extent by virtue of changing value (such as in the case of center-aligned text or the extent of a health bar).

[0089] This allows for predefined regions to be defined in the configuration file, as described elsewhere in this article.

[0090] However, in many games, one or more HUD elements are dynamic, meaning they move around the video game environment. This can simply mean that they bob up and down as part of a virtual head bob or other first-person perspective movement (e.g. to indicate an earthquake or other environmental event), or it can mean that they are more tightly integrated with the virtual environment. For example, bullet counts or weapon reloads can be integrated into the user's in-game weapon, or statistics for a player or non-player character can be positionally associated with the character such as a life bar located above their head or in a circle around their feet.

[0091] As mentioned previously, in the case of static HUD elements, the image analysis system typically looks for one or more HUD elements at fixed locations on the screen identified by a configuration file. The file identifies the region and typically the starting point of each HUD element.

[0092] Meanwhile, in the case of dynamic HUD elements, the image analysis system uses a two-step approach: first, the system finds the anchor point (which actually corresponds to the starting point of the area containing the HUD element); then, the image analysis system finds the HUD element in a manner similar to that previously described in this article. Thus, the anchor point is similar to the predetermined position of a static HUD element.

[0093] The number of anchor points required depends on the nature of the dynamic movement.

[0094] For example, when the entire HUD moves in response to the conceptual movement of a virtual camera providing a first-person perspective on the virtual environment (and therefore multiple HUD elements move in the same manner), the anchor point can involve a HUD element that is particularly easy to identify (both in terms of computational cost and / or disambiguation with respect to the general game environment) and is generally more reliably presented than other HUD elements. An example is a life bar, which generally has a predictable shape, color, and brightness compared to the background, and is more likely to be presented than other elements (such as an armor bar that can be completely depleted during normal gameplay); at least a first portion of the life bar can be detected in a manner similar to the image-type elements or temporary / conditional elements described elsewhere in this document; a pixel pattern of the starting point of the life bar can be searched for in an extended area containing the changing position of the life bar that moves due to head shaking or other effects (e.g., a green area with a white border on at least three sides assuming that the life bar is partially truncated). The search can be based on a template match between the position of the HUD element and the image, or based on a cross-correlation between the HUD element and the image area, or based on any more efficient method of searching for graphical elements in a wider image.

[0095] Once the corresponding portion of the health bar has been identified, this will serve as an anchor or reference point for the health bar and other HUD elements that have a fixed relationship to the health bar, but are similarly offset by the modified viewpoint.

[0096] Once the anchor point is determined, the various HUD elements may be identified in a suitable manner as described elsewhere herein.

[0097] In this case, the configuration file can provide information to help search for an anchor point (such as a pixel pattern at the start of a health bar) and an extended area to help search for the anchor point. The configuration file can then also provide the position of other HUD elements relative to this anchor point.

[0098] It should be understood that the use of positions relative to anchor points can also be used for static HUD elements to simplify or unify representation and processing within the image analysis system, but in this case the anchor points are predetermined. Similarly, in cases where HUD elements are static most of the time but occasionally dynamic (e.g., in cases where an occasional event causes camera shake), the system can assume a predetermined position, but if any HUD element (e.g., a health bar) becomes unreadable (e.g., because it moves out of the expected position), the image analysis system switches to a dynamic HUD strategy and actively searches for an anchor point.

[0099] As mentioned above, HUD elements can also be moved independently of each other, such as when they are associated with game elements (such as a user avatar's weapon or a non-player character).

[0100] In this case, each HUD element (or a subset of HUD elements that move together) has its own anchor point to search for the corresponding HUD element. The configuration file may again provide the region in which to search for the HUD.

[0101] So again refer to Figure 2 , assuming that HUD region 230 (boss health bar) is dynamic, the region moves around the screen with the boss. In this case, the HUD element to be searched would be a red line surrounded by a white border (i.e., similar to the part of the HUD most likely to be seen) located in the depicted area near the left-hand end of the boss health bar.

[0102] The profile may provide (relatively large) areas where dynamic HUD elements are located. Optionally, the profile may provide an indication of the relative probability of where dynamic HUD elements are located based on movement patterns or observed "heat maps" so that the system can prioritize its search in the most efficient way.

[0103] Once a HUD element is found and used as an anchor for the element and any associated elements identified by the configuration file (e.g., the boss' name in this case), it can be tracked in subsequent frames based on its position in the previous frame and optionally based on a motion vector (i.e., a short-term track of the direction of motion of the HUD element) to predict its position in the current frame. The HUD element can then be searched starting from or centered around this predicted position.

[0104] Thus, more generally, a dynamic HUD is searched in an image with reference to a representative HUD element, optionally within an area provided by a configuration file and / or predicted from a previous image frame. Once found, the position of the HUD element can be used as an anchor point for any HUDs that are stationary relative to the dynamic HUD (i.e., all HUDs move in the same way).

[0105] Post-processing

[0106] Whether static or dynamic, once a HUD element is detected, it can be analyzed and data extracted in the manner described previously in this article.

[0107] As described above, the data may include numerical values ​​corresponding to various on-screen states (e.g., life bars, etc.) so that the system can record a player's health (and / or any other such values, such as stamina, mana / energy, armor, weapon reload, possessions, etc., as appropriate).

[0108] The data may also include identification (whether by text or merely by graphics) of transient elements, such as equipped inventory items and enemies or other non-player characters encountered by the user, and optionally any on-screen states associated with those elements.

[0109] This data can be cleaned to account for any reading errors. For example, character names can be compared to a vocabulary to confirm that they have been read correctly, and it can be assumed that the names are consistent from one frame to the next so that any spelling errors can be corrected (while complete changes in written content can be detected as changes of a threshold magnitude).

[0110] More generally, it can be assumed that there is some degree of continuity in transient elements, such that any change below a threshold amount can be considered an error, and the nearest stable identification can be used.

[0111] At the same time (especially for dynamic HUDs), if the HUD element cannot be found within the available time (usually per frame), a certain degree of continuity can be assumed again and the most recent stable identification can be used. In this case, if the HUD is still not found after N frames (where Ni is a predetermined number), it can be assumed that the HUD element has disappeared, and optionally, the information can be updated to indicate that it disappeared before the first frame of N frames.

[0112] analyze

[0113] The data obtained from the above analysis can be used to derive important information about the game, gameplay and / or users that is independent of any telemetry data (if any) sent by the game itself.

[0114] Examples include:

[0115] Particularly effective hits on enemies: A large drop in an enemy's health bar indicates a critical hit or a particularly precise blow, the timestamp of a video clip of a boss fight can be associated with a number representing the hit value (incremental to the boss's health bar value), and the top N hit values ​​can be used to create a summary / highlight video of the user's best attacks on the boss.

[0116] When a player is on the verge of death in the game: A video clip of the game can be marked when the player's health reaches a threshold determined by the techniques herein, and the video can then be saved (or marked as saved) and continued to be marked until the health bar reaches zero, or the health jumps to a threshold amount (e.g., due to a last-minute health pack). At the same time, if neither of these events occurs within a predetermined time (e.g., because the player escapes from a threat and then survives a period of low health), the video clip can be unmarked, or the marking can be otherwise canceled (e.g., a counter-marking can be performed). In this way, a summary / highlight video of a player's death and / or near-death escape can be created.

[0117] More generally, any threshold level of a measurable user statistic, or a threshold change therein, can be used to trigger candidate recording of a video clip of the game (or, if the recording is automatic (e.g. on a circular buffer), marking the relevant portion of the recording as a candidate clip). So, in addition to near-death experiences, when their character levels up or wins a big prize could also be automatically recorded or marked for preservation.

[0118] It should be understood that when speculative recording is made on a circular buffer or similar area, the recording can be non-causal - that is, for example, if a keyword or phrase (such as "first place") is detected, then the video from N seconds ago can be marked for recording because it is likely that the player won the game. Similarly, when the name of a boss is detected, the video from N seconds ago can be marked for recording because the boss will usually appear prominently before the name is displayed on the screen. In these examples, it should be understood that the value of N may vary for different games or in-game events, and can be defined, for example, in a configuration file.

[0119] These clips can then be compiled into a highlight reel for users to upload or share, and / or used by the game as a review / memo for the user to re-familiarize themselves with the plot the next time they play the game. In this case, the clips in the reel can be sorted by importance (e.g., according to a scheme in the configuration file) so that the highlight reel is a filtered version of the clips, with the filtering based on the length of time since the user last played the game (e.g., if playing again the next day, only the most important clips are shown, if playing again next week, more clips are shown, and if playing again next month, all clips are shown).

[0120] However, the analysis need not be limited to a selection of video clips.

[0121] For example, analytics may be related to a user's play style (e.g., using spells that consume a lot of in-game magic resources, or always equipping melee or sniping items). This information may be useful to developers for balancing games or allocating resources to craft additional items, but more generally, this information may be useful, for example, to recommend other games that fit a player's style (which, in turn, may optionally depend at least in part on how many players in that game use a play style similar to the current user in that game).

[0122] Other analyses will be apparent to those skilled in the art. For example, detecting how often a player dies to get an idea of ​​their general skill level compared to the wider player population, or similarly detecting how long it takes to defeat a boss, and / or how much health they lose in the process. This information can then be used to provide system-based or in-game accessibility features, or again to recommend such features or games that have them (or whose difficulty level is comparable to the difficulty level the user is performing).

[0123] It should be appreciated that while the above techniques are described in the context of video games, they are not limited thereto. For example, a system could identify when a system is experiencing some form of slowdown by monitoring a progress bar without directly interacting with the associated code, thus enabling detection of issues regardless of what program is currently in use. Similarly, for systems related to healthcare, manufacturing / factory control, etc., where the system itself may be protected / hardcoded / outdated, or otherwise impractical to interact with at the data level, the techniques herein would enable third parties to monitor and analyze their UIs for significant events, as a source of additional warnings / analysis, and / or as a reliable black box.

[0124] Summary of Embodiments

[0125] Now go to Figure 3 In the generalized embodiment of the present specification, the image-based program state analysis method includes the following steps.

[0126] For a given image generated by a program (e.g. Figure 2 ), in a first step s310, identifying at least a first position within the image that is occupied by a corresponding user interface element (e.g., within a HUD), as described elsewhere herein—for example using information from a configuration file about where the user interface element is or is expected to be as predetermined points or areas, static or defining a dynamic range of possible positions, and / or using image analysis within the image to identify the position(s).

[0127] In a second step s320, predetermined points on corresponding user interface elements in the image are identified, as described elsewhere herein—for example, for one or more user interface elements, an anchor point such as the starting point of a life bar is identified, or an anchor point where an image or text element is located is identified.

[0128] In a third step s330, information conveyed by a respective user interface element is identified with reference to the predetermined point, as described elsewhere herein - for example by estimating the extension of a life bar, matching an image with a reference, reading (alpha)numeric text, etc.

[0129] And, in a fourth step s340, outputting the information for subsequent analysis, as described elsewhere herein - for example, identifying and optionally sorting video clips to compile highlights or summaries for a user, or to provide selection and performance trends for a user or a corpus of users.

[0130] It will be apparent to those skilled in the art that variations of the above methods corresponding to the operation of the various embodiments of the apparatus described and claimed herein are considered to be within the scope of the present invention, including but not limited to:

[0131] - the first position or an area including the first position is provided by a configuration file of the program, as described elsewhere herein;

[0132] -predetermined points on the respective user interface elements are static within the programmatically generated image, as described elsewhere herein;

[0133] - a predetermined point on a corresponding user interface element is moved within an image generated by the program, as described elsewhere herein;

[0134] - the step of identifying a predetermined point on the corresponding user interface element in the image comprises: performing image analysis to identify an image element corresponding to at least a first portion of the corresponding user interface element, and identifying a corresponding position of the predetermined point on the corresponding user interface element in the image based on the position of the image element, as described elsewhere herein;

[0135] - In this case, optionally, the given image includes a plurality of further user interface elements that move in the same manner as the first corresponding user interface element, and the step of identifying the predetermined point on the respective further user interface elements includes: identifying the predetermined point on one or more of the further corresponding user interface elements in the image as a respective predetermined relative offset relative to the identified position of the predetermined point on the first corresponding user interface element, as described elsewhere herein

[0136] as stated;

[0137] - the respective user interface element is a bar element, and the step of identifying information comprises: performing image analysis to detect how many of the largest bars originating from a predetermined point are currently depicted within the image, and estimating the attribute value based on the number currently depicted, as described elsewhere herein;

[0138] - the corresponding user interface element is an image-type element, and the step of identifying the information comprises: performing image analysis to identify the image element within a defined area relative to the predetermined point by: comparing a representation of the image-type element within the defined area to corresponding representations of a plurality of reference image elements; and obtaining an item definition associated with a best matching reference item, as described elsewhere herein;

[0139] - the corresponding user interface element is one or more numeric character elements, and the step of identifying the information comprises: performing image analysis using optical character recognition to identify the one or more numeric character elements within a defined area relative to a predetermined point, as described elsewhere herein;

[0140] - the respective user interface element is one or more alphanumeric character elements, and the step of identifying the information comprises: performing image analysis using optical character recognition to identify the one or more alphanumeric character elements within a defined area relative to a predetermined point, as described elsewhere herein;

[0141] - In this case, the step of performing image analysis comprises employing a search strategy for said alphanumeric character elements in response to the expected alignment of text within said defined area, as described elsewhere herein

[0142] Place described;

[0143] - the step of identifying information does not include identifying text, as described elsewhere in this document;

[0144] - the step of identifying the information delivered by the respective user element is performed at a frequency responsive to the type of said information, as described elsewhere herein;

[0145] -Subsequent analysis based on a given image or sequence of such images includes one or more selected from a list consisting of: identifying events based on changes in the identified information (e.g., damage inflicted by a boss or points earned by a player); identifying events based on thresholds associated with the identified information (e.g., health points falling below a threshold); identifying trends based on the identified information (e.g., player style, preferences, and / or abilities); and identifying states based on the identified information (e.g., progress of a match completed or delayed operation), as described elsewhere herein.

[0146] It will be appreciated that the above methods may be performed on conventional hardware suitably adapted by means of software instructions or by the inclusion or replacement of dedicated hardware.

[0147] Thus, the adaptation of existing portions of the device required to configure it as at least a portion of an image-based program state analysis system may be implemented at least in part in the form of a computer program product comprising processor executable instructions stored on a non-transitory machine-readable medium (e.g., a floppy disk, an optical disk, a hard disk, a solid-state drive, a PROM, a RAM, a flash memory, or any combination of these or other storage media), or implemented in hardware as an ASIC (application specific integrated circuit) or an FPGA (field programmable gate array) or other configurable circuit suitable for adapting a conventional equivalent device. In addition, such a computer program may be transmitted via a data signal over a network (e.g., an Ethernet, a wireless network, the Internet, or any combination of these or other networks).

[0148] Therefore, in the summary embodiments of this specification, reference is made again to Figure 1 , an image-based program state analysis system (eg, an entertainment system 10 (such as ))include:

[0149] An image processor configured (e.g., by suitable software instructions) to, for a given image generated by a program (e.g., a game, healthcare or factory monitoring, or other suitable program):

[0150] identifying at least a first position within the image occupied by a corresponding user interface element;

[0151] identifying a predetermined point on a corresponding user interface element in the image;

[0152] identifying, with reference to the predetermined point, information conveyed by the corresponding user interface element; and

[0153] Output information for subsequent analysis.

[0154] Examples of this generalized embodiment that implement the methods and techniques described herein (eg, by using appropriate software instructions) are contemplated within the scope of the present application, including but not limited to those listed in the preceding generalized embodiments.

[0155] The above discussion discloses and describes only exemplary embodiments of the present invention. As will be appreciated by those skilled in the art, the present invention may be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the disclosure of the present invention is intended to be illustrative, rather than limiting, the scope of the present invention and other claims. The present disclosure (including any readily discernible variations taught herein) partially defines the scope of the appended claim terms so that no inventive subject matter is dedicated to the public.

Claims

1. A method for analyzing program status based on an image, comprising the following steps: For a given image generated by the program, identifying at least a first position within the image occupied by a corresponding user interface element; identifying a predetermined point on the corresponding user interface element in the image; identifying information conveyed by the corresponding user interface element with reference to the predetermined point; as well as The information is output for subsequent analysis.

2. The method according to claim 1, wherein: The first position or an area including the first position is provided by a configuration file of the program.

3. The method according to claim 1, wherein: The predetermined point on the corresponding user interface element is static within an image generated by the program.

4. The method according to claim 1 or 2, wherein: The predetermined point on the corresponding user interface element moves within an image generated by the program.

5. The method according to claim 4, wherein: The step of identifying a predetermined point on the corresponding user interface element in the image comprises: performing image analysis to identify an image element corresponding to at least a first portion of the corresponding user interface element; and A corresponding position of the predetermined point on the corresponding user interface element in the image is identified based on the position of the image element.

6. The method according to claim 5, wherein: The given image includes a plurality of further user interface elements that move in the same manner as the first corresponding user interface element, and the step of identifying predetermined points on the corresponding further user interface elements includes: A predetermined point on one or more of the additional respective user interface elements in the image is identified as a respective predetermined relative offset relative to the identified position of the predetermined point on the first respective user interface element.

7. A method according to any one of the preceding claims, wherein: The corresponding user interface element is a bar element, and the step of identifying the information comprises: performing image analysis to detect how many largest bars originating from the predetermined point are currently depicted within the image; and Estimates the value of a property based on the quantity currently being depicted.

8. A method according to any one of the preceding claims, wherein: The corresponding user interface element is an image type element, and the step of identifying the information comprises: Image analysis is performed to identify image elements within a defined area relative to the predetermined point by: comparing the representation of the image type element within the defined area with corresponding representations of a plurality of reference image elements; and Get the item definition associated with the best matching reference item.

9. A method according to any one of the preceding claims, wherein: The corresponding user interface element is one or more numeric character elements, and the step of identifying the information comprises: Image analysis is performed using optical character recognition to identify one or more digital character elements within a defined area relative to the predetermined point.

10. A method according to any one of the preceding claims, wherein: The corresponding user interface element is one or more alphanumeric character elements, and the step of identifying the information comprises: Image analysis is performed using optical character recognition to identify one or more alphanumeric character elements within a defined area relative to the predetermined point.

11. The method according to claim 10, wherein: The step of performing image analysis includes employing a search strategy for the alphanumeric character elements in response to the expected text alignment within the defined area.

12. The method according to any one of claims 1 to 8, wherein: The step of identifying the information does not include identifying text.

13. A method according to any one of the preceding claims, wherein: The step of identifying said information delivered by the corresponding user element is performed at a frequency responsive to the type of said information.

14. A method according to any one of the preceding claims, wherein: Subsequent analysis based on a given image or a sequence of such images comprises one or more selected from the list consisting of: i. Identify events based on changes in identified information; ii. identifying an event based on a threshold associated with the identified information; iii. Identify trends based on the identified information; as well as iv. Identify the state based on the identified information.

15. A computer program comprising computer executable instructions adapted to cause a computer system to perform the method of any preceding claim.

16. An image-based program state analysis system, comprising: An image processor, the image processor being configured to: for a given image generated by the program: identifying at least a first position within the image occupied by a corresponding user interface element; identifying a predetermined point on the corresponding user interface element in the image; identifying information conveyed by the corresponding user interface element with reference to the predetermined point; as well as The information is output for subsequent analysis.

Images