Front-end application error positioning method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a front-end application program error positioning method, which comprises the following steps of: analyzing a program code in a code warehouse of a front-end application program, and determining a characteristic graph of each interface component related to the program code and a style inheritance graph of each interface component; receiving a page screenshot corresponding to the front-end application program; identifying an abnormal interface component in the page screenshot, and extracting structure information and abnormal information corresponding to the abnormal interface component; and positioning an error program code segment corresponding to the abnormal interface component from the program code according to the structure information and the abnormal information corresponding to the abnormal interface component, the characteristic atlas of each interface component and the style inheritance atlas of each interface component. Through the method, the error program code segment in the front-end application program can be positioned based on the page screenshot containing the abnormal interface component. The invention further provides a front-end application error positioning device, electronic equipment, a computer readable storage medium and a computer program product.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technologies, and in particular, to a method, an apparatus, an electronic device, a storage medium, and a program product for locating errors in a front-end application program. Background Art

[0002] In the process of software development, the location and repair of program errors are very time-consuming and costly. At present, there are still many difficulties in locating errors in front-end application programs. For example, the front-end application program error detection tools in related technologies rely on users to manually associate page anomalies with code logic and cannot automatically map the user interface anomalies in the user interface screenshots or screen recording videos submitted by users to specific program codes. In addition, the front-end application program error location solutions in related technologies can only capture error stacks and lack the ability to completely reproduce the user operation chain. Further, the front-end application program error location solutions in related technologies are limited to line-level error location and cannot understand the hierarchical relationship between interface components, resulting in fragmentation of program repair solutions. Summary of the Invention

[0003] In view of this, embodiments of the present disclosure provide a method, an apparatus, an electronic device, a storage medium, and a program product for locating errors in a front-end application program, which can solve or partially solve the above problems to a certain extent.

[0004] In some embodiments of the present disclosure, the method for locating errors in a front-end application program according to the embodiments of the present disclosure may include: parsing program codes in a code repository of the front-end application program to determine feature maps of each interface component involved in the program codes and style inheritance maps of each interface component; receiving a page screenshot corresponding to the front-end application program; identifying an abnormal interface component in the page screenshot, and extracting structure information and abnormal information corresponding to the abnormal interface component; and locating an error program code segment corresponding to the abnormal interface component from the program codes according to the structure information and abnormal information corresponding to the abnormal interface component, the feature maps of each interface component, and the style inheritance maps of each interface component.

[0005] In some embodiments of the present disclosure, the feature maps of each interface component include: identifiers of each interface component, attribute modes of each interface component, status dependency information of each interface component, and event binding information of each interface component; and the style inheritance maps of each interface component include: inheritance relationships between styles of each interface component.

[0006] In some embodiments of the present disclosure, parsing the program code in the code repository of the front-end application to determine the feature map of each interface component and the style inheritance map of each interface component includes: parsing the script program code in the code repository of the front-end application to determine the feature map of each interface component; and parsing the component style program code in the code repository of the front-end application to determine the style inheritance map of each interface component.

[0007] In some embodiments of the present disclosure, parsing the script program code in the code repository of the front-end application to determine the feature map of each interface component includes: converting the script program code into an abstract syntax tree; identifying at least one component definition node in the abstract syntax tree; for each component definition node in the at least one component definition node, extracting the component name of the component definition node as the identifier of an interface component; determining the attribute mode of the interface component based on the metadata of the attribute parameters associated with the component definition node; determining the state dependency information of the interface component based on the declaration and use of the state variables associated with the component definition node; determining the event binding information of the interface component based on the event handling logic associated with the component definition node; and generating the feature map of the interface component based on the identifier of the interface component, the attribute mode of the interface component, the state dependency information of the interface component, and the event binding information of the interface component.

[0008] In some embodiments of the present disclosure, parsing the component style program code in the code repository of the front-end application to determine the style inheritance map of each interface component includes: determining the hierarchical relationship between each interface component based on the nesting relationship of the component style program code; for each interface component in each interface component, extracting the style attributes of the interface component based on the component style program code, and in response to determining that the interface component sets an inheritable style attribute and does not override the inheritable style attribute, determining the parent component of the interface component based on the hierarchical relationship, and determining the inheritance relationship from the parent component to the interface component corresponding to the inheritable style attribute; and generating the style inheritance map of each interface component based on the inheritance relationship corresponding to the inheritable style attribute.

[0009] In some embodiments of the present disclosure, identifying the abnormal interface components in the page screenshot and extracting the structure information and abnormal information corresponding to the abnormal interface components include: performing layout parsing on the page screenshot to extract the structure information corresponding to at least one interface component to be detected included in the page screenshot; identifying the abnormal interface components from the at least one interface component to be detected based on the interface component library associated with the program code and the structure information of the at least one interface component to be detected; and determining the structure information and abnormal information corresponding to the abnormal interface components.

[0010] In some embodiments of the present disclosure, identifying the abnormal interface components from the at least one interface component to be detected based on the interface component library associated with the program code and the structure information of the at least one interface component to be detected includes: determining at least one first target interface component corresponding to the at least one interface component to be detected from the interface component library based on the structure information corresponding to the at least one interface component to be detected; extracting the metadata of the at least one first target interface component from the interface component library; inputting the metadata of the at least one first target interface component into an image generation model to generate an expected page image; comparing the expected page image with the page screenshot to determine the abnormal display area and the display difference information corresponding to the abnormal display area; and determining the interface component to be detected corresponding to the abnormal display area as the abnormal interface component; and determining the structure information and abnormal information corresponding to the abnormal interface components includes: extracting the structure information corresponding to the abnormal interface component; and determining the display difference information corresponding to the abnormal display area as the abnormal information corresponding to the abnormal interface component.

[0011] In some embodiments of the present disclosure, locating the error program code segment corresponding to the abnormal interface component from the program code according to the structure information and abnormal information corresponding to the abnormal interface component, the feature maps of the respective interface components, and the style inheritance maps of the respective interface components includes: matching the structure information corresponding to the abnormal interface component with the feature maps of the respective interface components to determine a second target interface component corresponding to the abnormal interface component; locating the target program code corresponding to the second target interface component in the program code; and locating the error program code segment corresponding to the abnormal information in the target program code based on the abnormal information corresponding to the abnormal interface component, the feature maps of the respective interface components, and the style inheritance maps of the respective interface components.

[0012] In some embodiments of the present disclosure, parsing the program code in the code repository of the front-end application to determine the feature map of each interface component involved in the program code and the style inheritance map of each interface component includes: parsing the program code based on a large language model to determine the feature map of each interface component involved in the program code and the style inheritance map of each interface component; identifying the abnormal interface components in the page screenshot and extracting the corresponding structure information and abnormal information of the abnormal interface components includes: identifying the abnormal interface components in the page screenshot based on a multimodal large model and extracting the corresponding structure information and abnormal information of the abnormal interface components; and locating the error program code segment corresponding to the abnormal interface component from the program code according to the structure information and abnormal information corresponding to the abnormal interface component, the feature map of each interface component, and the style inheritance map of each interface component includes: locating the error program code segment corresponding to the abnormal interface component from the program code based on the large language model according to the structure information and abnormal information corresponding to the abnormal interface component, the feature map of each interface component, and the style inheritance map of each interface component.

[0013] In some embodiments of the present disclosure, receiving the page screenshot corresponding to the front-end application includes: receiving the screen recording video corresponding to the front-end application; extracting at least one key image frame from the screen recording video; and using each key image frame as the page screenshot respectively.

[0014] In some embodiments of the present disclosure, the above front-end application error location method further includes: determining a user operation sequence corresponding to the screen recording video based on the at least one key image frame; wherein the user operation sequence includes: operation type and the object being operated; determining a third target interface component corresponding to the object being operated and the abnormal state of the third target interface component based on the user operation sequence and the feature map of each interface component; and determining the error program code segment from the program code based on the abnormal state of the third target interface component and the feature map of each interface component.

[0015] In some embodiments of the present disclosure, determining a third target interface component corresponding to the operated object and an abnormal state of the third target interface component based on the user operation sequence and the feature maps of the respective interface components includes: converting the user operation sequence into an event stream; matching the event stream with the event binding information of the operated object in the feature maps of the respective interface components to determine the third target interface component corresponding to the operated object and restore the event handling logic triggered by the user operation; determining a state transfer path of the third target interface component based on the event handling logic and the state dependency information of the third target interface component in the feature maps of the respective interface components; and determining the abnormal state of the third target interface component based on the state transfer path of the third target interface component.

[0016] Corresponding to the above front-end application error localization method, embodiments of the present disclosure also disclose a front-end application error localization device, including:

[0017] A map generation module, configured to parse program codes in a code repository of a front-end application to determine feature maps of respective interface components involved in the program codes and style inheritance maps of the respective interface components;

[0018] An image receiving module, configured to receive a page screenshot corresponding to the front-end application;

[0019] An abnormal interface component recognition module, configured to recognize an abnormal interface component in the page screenshot and extract structure information and abnormal information corresponding to the abnormal interface component; and

[0020] A localization module, configured to locate an error program code segment corresponding to the abnormal interface component from the program codes according to the structure information and abnormal information corresponding to the abnormal interface component, the feature maps of the respective interface components, and the style inheritance maps of the respective interface components.

[0021] In addition, embodiments of the present disclosure also provide an electronic device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, where when the processor executes the computer program, the above front-end application error localization method is implemented.

[0022] Embodiments of the present disclosure also provide a non-transitory computer-readable storage medium, where the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to cause a computer to execute the above front-end application error localization method.

[0023] Embodiments of the present disclosure also provide a computer program product, including computer program instructions, which, when running on a computer, cause the computer to execute the above-mentioned front-end application program error location method.

[0024] As can be seen, the above-mentioned front-end application program error location method, device, electronic device, storage medium, and program product can parse the program code in the front-end application program code repository to determine the feature map of each interface component involved in the front-end application program and the style inheritance map of each interface component, and after identifying the abnormal interface components in the page screenshot, locate the error program code segment corresponding to the abnormal interface component in the program code of the front-end application program based on the feature map of each interface component and the style inheritance map of each interface component, so as to quickly and effectively locate the error program code segment in the front-end application program. In addition, in some embodiments, since the style inheritance map of each interface component described above describes the inheritance relationship of the interface component styles and reflects the hierarchical relationship between the interface components, the front-end application program error location method described in the embodiments of the present disclosure can trace the program code errors of the upper-level interface components, thereby making the error location more accurate and effective. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the following will briefly introduce the drawings required for use in the embodiments or related technology descriptions. Obviously, the drawings described below are only the embodiments of the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0026] Figure 1 Shows an application scenario of the front-end application program error location method described in some embodiments of the present disclosure.

[0027] Figure 2 Shows the implementation process of the front-end application program error location method described in some embodiments of the present disclosure.

[0028] Figure 3 Shows the implementation process of the method for parsing the script program code in the code repository of the front-end application program to determine the feature map of each interface component described in some embodiments of the present disclosure.

[0029] Figure 4 Shows the implementation process of the method for parsing the component style program code in the code repository of the front-end application program to determine the style inheritance map of each interface component described in some embodiments of the present disclosure.

[0030] Figure 5Shows the implementation process of a method for identifying abnormal interface components in a screenshot of an identification page and extracting the structure information and abnormal information corresponding to the abnormal interface components according to some embodiments of the present disclosure.

[0031] Figure 6 Shows the implementation process of a method for identifying abnormal interface components from at least one interface component to be detected based on an interface component library associated with program code and the structure information of at least one interface component to be detected according to some embodiments of the present disclosure.

[0032] Figure 7 Shows the implementation process of a method for locating an error program code segment corresponding to an abnormal interface component in program code according to the structure information and abnormal information corresponding to the abnormal interface component, the feature map of each interface component, and the style inheritance map of each interface component according to some embodiments of the present disclosure.

[0033] Figure 8 Shows the implementation process of a method for locating a state transfer error of an interface component in a front-end application program according to some embodiments of the present disclosure.

[0034] Figure 9 Shows the internal structure of a front-end application program error location device according to some embodiments of the present disclosure.

[0035] Figure 10 Illustrates a more specific schematic diagram of the hardware structure of an electronic device according to some embodiments of the present disclosure. Detailed implementation manners

[0036] To make the objectives, technical solutions, and advantages of the present disclosure clearer and more understandable, the present disclosure will be further described in detail below with reference to specific embodiments and the accompanying drawings.

[0037] It should be noted that unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should have the ordinary meanings understood by those of ordinary skill in the field to which the present disclosure belongs. The "first", "second", and similar terms used in the embodiments of the present disclosure do not denote any order, quantity, or importance, but are only used to distinguish different components. The terms such as "including" or "comprising" mean that the elements or objects appearing before this term cover the elements or objects listed after this term and their equivalents, without excluding other elements or objects. The terms such as "connected" or "coupled" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The terms such as "upper", "lower", "left", and "right" are only used to represent relative positional relationships, and when the absolute position of the object being described changes, the relative positional relationship may also change accordingly.

[0038] It is understandable that before using the technical solutions of the various embodiments of the present disclosure, the types, usage scopes, usage scenarios, etc. of the personal information involved will be informed to users in an appropriate manner, and user authorization will be obtained.

[0039] For example, when responding to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested by them will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application program, server, or storage medium that performs the operations of the technical solutions of the present disclosure based on the prompt message.

[0040] As an optional but non-limiting implementation manner, the way of sending a prompt message to the user in response to receiving an active request from the user can be, for example, in the form of a pop-up window. The prompt message can be presented in text form in the pop-up window. In addition, the pop-up window can also carry selection controls for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0041] It is understandable that the above process of notifying and obtaining user authorization is only illustrative and does not limit the implementation manner of the present disclosure. Other ways that comply with relevant laws and regulations can also be applied to the implementation manner of the present disclosure.

[0042] For the sake of clarity in description, before describing the specific technical solutions of the embodiments of the present disclosure, several technical terms involved in the embodiments of the present disclosure are first explained.

[0043] An application program (Application, abbreviated as APP), simply referred to as an application, can refer to a computer program for completing one or more specific tasks. It runs in user mode, can interact with users, and has a visible user interface. Among them, the development of an application program can include two main parts: front-end development and back-end development. Front-end development is generally responsible for the user interface and interaction experience of the application program, and back-end development is generally responsible for the server-side logic and database management of the application program. The front-end application program in the embodiments of the present disclosure can refer to the application program corresponding to front-end development, and in some special cases, it can also include the application program corresponding to back-end development.

[0044] A large language model (Large Language Model, abbreviated as LLM) can refer to a neural network model used for natural language processing tasks and having a large number of parameters and a complex structure. The LLM is trained by using a large amount of data and uses deep learning techniques to learn the basic patterns and structures of language. Compared with small models, the LLM can achieve higher performance and usually has better generalization ability in various tasks.

[0045] A multimodal large model can refer to a neural network model with a large number of parameters and a complex structure that jointly trains multimodal information such as text, images, videos, and audio.

[0046] A prompt can be an injection instruction used to "command" a large language model or a multimodal large model to think about problems and output content according to a preset idea. A prompt is an instruction or information that guides or triggers a large language model or a multimodal large model to make a response.

[0047] A code repository can be a management system that centrally stores source code and related resources, and can be used to save, share, and version control code. Currently, code repositories provide tools for collaborative development, code review, and project management for development teams, and are one of the indispensable infrastructures in modern software development.

[0048] An interface component can refer to a reusable and well - encapsulated user interface component created in front - end application development, thereby improving the maintainability and scalability of front - end application code. Currently, commonly used interface components in front - end application development include: navigation components, carousel components, table components, form components, pop - up components, image display components, modal box components, drop - down menu components, label components, date selection components, and button components, etc.

[0049] Static code analysis can refer to a technique that, without actually executing a program, detects potential errors, security vulnerabilities, and coding standard issues by analyzing code structure, syntax, and semantics.

[0050] An Abstract Syntax Tree (AST) can be a tree - like representation of the abstract syntax structure of source code, or the core intermediate representation form of static code analysis.

[0051] Cascading Style Sheets (CSS) is responsible for the presentation and layout of web pages. CSS can not only statically decorate web pages, but also dynamically format various elements of web pages in cooperation with various scripting languages. In CSS, style inheritance is a very important concept. Style inheritance allows certain property values to be passed from a parent component to a child component. This means that if a component does not specify a value for a certain property, it will inherit the corresponding property value of its parent component. Through the style inheritance mechanism, styles can be easily shared throughout the page or component without having to repeat the same style code for each component. It should be noted that not all CSS properties can be inherited. Generally, text - related properties and some visual formatting properties can be inherited.

[0052] As described above, there are still many difficulties in locating errors in current front-end applications, which remains a very time-consuming and costly part of software development. To this end, embodiments of the present disclosure provide a method for locating errors in front-end applications.

[0053] Figure 1 Fig. 4 shows an application scenario of the method for locating errors in front-end applications according to embodiments of the present disclosure. As Figure 1 shown, the method for locating errors in front-end applications according to embodiments of the present disclosure can be applied to a front-end application error location device 100. In a specific application, the above-mentioned front-end application error location device 100 can be implemented based on a large language model and / or a multimodal large model. For a certain front-end application, when the user 200 discovers an abnormal user interface during the development process, testing process, or usage process of the above-mentioned front-end application, a page screenshot or a screen recording video related to the discovered abnormal user interface can be input into the above-mentioned front-end application error location device 100. Next, the above-mentioned front-end application error location device 100 completes the recognition of abnormal interface components in the page screenshot or screen recording video submitted by the user based on the method for locating errors in front-end applications given in embodiments of the present disclosure, and locates the relevant error program code segments in the code repository 300 of the front-end application based on the recognized abnormal interface components, thereby effectively and accurately realizing the location of error program code segments in the front-end application.

[0054] Next, the method for locating errors in front-end applications executed by the above-mentioned front-end application error location device 100 in embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0055] Figure 2 Fig. 5 shows the implementation process of the method for locating errors in front-end applications according to embodiments of the present disclosure. As Figure 2 shown, the above-mentioned method for locating errors in front-end applications may include the following multiple steps.

[0056] In step 210, the program code in the code repository of the front-end application is parsed to determine the feature map of each interface component involved in the program code and the style inheritance map of each interface component.

[0057] In step 220, a page screenshot corresponding to the front-end application is received.

[0058] In step 230, the abnormal interface components in the page screenshot are recognized, and the structure information and abnormal information corresponding to the abnormal interface components are extracted.

[0059] In step 240, based on the structure information and exception information corresponding to the abnormal interface component, the feature map of each interface component, and the style inheritance map of each interface component, locate the error program code segment corresponding to the abnormal interface component from the program code.

[0060] The above front-end application error location method can parse the program code in the code repository of the front-end application to determine the feature map of each interface component involved in the front-end application and the style inheritance map of each interface component. After identifying the abnormal interface component in the page screenshot, based on the above feature map of each interface component and the style inheritance map of each interface component, locate the error program code corresponding to the abnormal interface component in the program code of the front-end application, so as to effectively locate the error program code segment in the front-end application.

[0061] Next, the specific implementation methods of each step of the above front-end application error location method will be further described in detail in combination with the accompanying drawings and specific embodiments.

[0062] Regarding the above step 210, in the embodiments of the present disclosure, the code repository of the front-end application is used to store the relevant source code and related resources of the front-end application. In particular, in the embodiments of the present disclosure, the program code of the front-end application stored in the code repository can be program code written based on different front-end web development technology stacks. For example, the above front-end web development technology stack can specifically refer to the React technology stack (a library or framework for building user interfaces) or the Vue technology stack (a framework for building user interfaces), etc. It can be seen that the embodiments of the present disclosure do not limit the technology stack used for front-end software development. That is to say, the front-end application error location method described in the embodiments of the present disclosure can be compatible with different technology stacks when applied, that is, it can locate the error program code segment in the front-end application across technology stacks.

[0063] In addition, in the embodiments of the present disclosure, the feature map of each interface component may include: the identifier of each interface component, the attribute mode of each interface component, the state dependency information of each interface component, and the event binding information of each interface component. The style inheritance map of each interface component includes: the inheritance relationship between the styles of each interface component. In application, the feature map of each interface component and the style inheritance map of each interface component can both be represented or stored by formatted data. In addition, the feature map of each interface component and the style inheritance map of each interface component can generally be determined by static code analysis of the front-end application.

[0064] Specifically, the identifier of the above interface component represents the identifier information that can uniquely identify an interface component in the front-end application, and can also be referred to as the identity identifier (Identity, ID) of the interface component. For example, the name of the interface component, etc. Usually, the identifier of the interface component can be obtained by analyzing the definition or naming rule of the interface component. The above property pattern of the interface component is usually used to represent the type and structure of the properties of the interface component. Usually, the property pattern of the interface component can be obtained by analyzing the types or definitions of the respective property parameters of the component. For example, by analyzing the property parameters (props) of the interface component defined by the -Props Schema (a schema for defining the properties of the interface component) based on the Vue technology stack, the property pattern of the interface component can be obtained. The above state dependency information (State Dependencies) of the interface component involves the declaration and use of internal state variables of the interface component, and is used to track the changes of the state variables involved in the interface component in different event handling logics. The above event binding information (Event Bindings) of the interface component is used to identify the event handling logic bound to the interface component. In this way, by using the feature map containing the above information to locate the errors in the front-end application, the location of the error program code segment in the front-end application can be more effectively achieved.

[0065] Specifically, in some embodiments of the present disclosure, the parsing of the program code in the code repository of the front-end application in step 210 to determine the feature map of each interface component and the style inheritance map of each interface component may specifically include the following two steps: parsing the script program code in the code repository of the front-end application to determine the feature map of each interface component; and parsing the component style program code in the code repository of the front-end application to determine the style inheritance map of each interface component. It should be noted that the execution order of the above two steps is not limited by the front-back order described in the above text, that is, the above two steps can be serially executed in any order or can also be executed in parallel.

[0066] It can be understood that the program code in the code repository of the front-end application can include script program code associated with the application execution logic, and can also include component style program code related to the interface component styles. By analyzing different types of code separately according to the different functions of the program code, not only can the feature maps of each interface component and the style inheritance maps of each interface component be determined respectively, but also the overall workload of code parsing can be reduced, thereby improving the execution efficiency of the method. Specifically, in the embodiments of the present disclosure, the above-mentioned script program code can be Java Script (JS) code or TypeScript (TS) code. In addition, in the embodiments of the present disclosure, the above-mentioned component style program code can be CSS code or style program code written through a CSS preprocessing language.

[0067] Figure 3 shows the implementation process of the method for parsing the script program code in the code repository of the front-end application and determining the feature maps of each interface component according to the embodiments of the present disclosure. As Figure 3 shown, the method for determining the feature maps of each interface component can specifically include the following multiple steps.

[0068] In step 310, convert the script program code into an abstract syntax tree.

[0069] In the embodiments of the present disclosure, the above-mentioned script program code can be converted into an abstract syntax tree through a static code analysis method.

[0070] In step 320, identify at least one component definition node in the abstract syntax tree.

[0071] In the embodiments of the present disclosure, the above-mentioned component definition node refers to a syntax tree node used to describe the logic and structure of the interface component involved in the above-mentioned script program code, and usually corresponds to structures such as function declarations, class declarations, or object expressions in the script program code. Usually, the component definition node can contain one or a combination of the following data: component name, component type identifier, attributes and parameters, logic and options, sub-components and content, and dynamic bindings and directives, etc.

[0072] In step 330, for each component definition node in the above-mentioned at least one component definition node, extract the component name of the above-mentioned component definition node as the identifier of an interface component.

[0073] In step 340, determine the attribute mode of the above-mentioned interface component based on the metadata of the attribute parameters associated with the above-mentioned component definition node.

[0074] Specifically, in the embodiments of the present disclosure, the type definition or default value of the attribute parameters associated with the component definition node can be parsed to obtain the attribute mode of the above-mentioned interface component. For example, in TS, interfaces or type aliases can be used to define the structure of the attribute parameters, so these information can be extracted as the attribute mode of the above-mentioned interface component.

[0075] In step 350, based on the declarations and uses of the state variables associated with the above-mentioned component definition node, the state dependency information of the above-mentioned interface component is determined.

[0076] Specifically, in the embodiments of the present disclosure, the declarations and uses of the state variables associated with the above-mentioned component definition node can be identified, so as to determine which event handling logics (e.g., event handling functions) reference or modify the state variables, thereby tracing the dependency relationship between the state variables and the event handling logics. It can be seen that in the embodiments of the present disclosure, the above-mentioned state dependency information mainly reflects the dependency relationship between the state variables of an interface component and the event handling logics.

[0077] In step 360, based on the event handling logics associated with the above-mentioned component definition node, the event binding information of the above-mentioned interface component is determined.

[0078] Specifically, in the embodiments of the present disclosure, the binding positions of the event handlers associated with the above-mentioned component definition node can be found, and the event handling logics corresponding to the events are associated, so as to determine the event handling logics (e.g., event handling functions) associated with the events, and further determine the event binding information of the interface component. It can be seen that in the embodiments of the present disclosure, the above-mentioned event binding information mainly reflects the binding relationship between specific events and the event handling logics. It can be seen that through the state dependency information and the event binding information, it can be determined how the state variables of the interface component will change when a certain event of a certain interface component occurs.

[0079] In step 370, based on the identifier of the above-mentioned interface component, the attribute mode of the above-mentioned interface component, the state dependency information of the above-mentioned interface component, and the event binding information of the above-mentioned interface component, the feature map of the above-mentioned interface component is generated.

[0080] In some embodiments of the present disclosure, as described above, the feature map of the above interface component can be represented or stored using structured data. For example, the above structured data can be data in the JavaScript Object Notation (JSON) format. It can be understood that representing or storing the feature map of the above interface component using structured data can standardize the data representation form, facilitate data reading and use, thereby effectively improving the overall execution efficiency and scalability of the method. Moreover, for front-end application programs developed through different front-end web development technology stacks, the features of interface components can be expressed through unified structured data, providing a basis for error location of front-end application programs across technology stacks.

[0081] In addition, as described above, in the style program of a component (such as CSS code), there may be an inheritance relationship between certain attribute values of a parent interface component and a child interface component, and in the style program of the component, it is clearly defined whether the attribute value of a certain interface component is inherited from the attribute value of its parent interface component or overrides the attribute value of its parent interface component. Based on this, Figure 4 shows the implementation process of the method for parsing the component style program code in the code repository of a front-end application program according to an embodiment of the present disclosure to determine the style inheritance map of each interface component. As Figure 4 shown, the method for determining the style inheritance map of each interface component may specifically include the following multiple steps.

[0082] In step 410, determine the hierarchical relationship between interface components based on the nesting relationship of the component style program code.

[0083] In step 420, for each of the above interface components, extract the style attributes of the above interface component based on the component style program code. In response to determining that the above interface component sets an inheritable style attribute and does not override the above inheritable style attribute, determine the parent component of the above interface component based on the above hierarchical relationship, and determine that there is an inheritance relationship from the above parent component to the above interface component for the above inheritable style attribute.

[0084] In step 430, generate the style inheritance map of each interface component based on the above inheritance relationship for the above inheritable style attribute.

[0085] Through the above method, after determining the inheritance relationship for each inheritable style attribute of each interface component, a style inheritance graph for inheritance among various interface components can be determined for each inheritable style attribute. After summarization, the style inheritance graphs of the above-mentioned various interface components can be obtained. The style inheritance graphs of the above-mentioned various interface components contain the inheritance relationships of all inheritable style attributes for inheritance among various interface components.

[0086] In an embodiment of the present disclosure, the above-mentioned style inheritance graphs of the various interface components can also be represented or stored using structured data. For example, JSON format can be used for data representation or storage. Similarly, representing or storing the style inheritance graphs of the above-mentioned interface components using structured data can not only standardize the data presentation form, facilitate data reading and usage, effectively improve the overall execution efficiency and scalability of the method, but also provide a basis for implementing front-end application error localization across technology stacks.

[0087] In some embodiments of the present disclosure, the above step 210 and its specific implementation method can be implemented through a large language model. It can be understood that implementing the solution of parsing the program code in the code repository of the front-end application to determine the feature graphs of the various interface components and the style inheritance graphs of the various interface components described in the above step 210 through a large language model can make full use of the advantages of the large language model in semantic understanding, can efficiently and accurately determine the feature graphs of the various interface components and the style inheritance graphs of the various interface components involved in the program code, thereby improving the effectiveness and accuracy of the above front-end application error localization method, and at the same time greatly reducing the development cost of the above front-end application error localization solution.

[0088] In a specific application, the key to using a large language model to perform the above-mentioned step 210 lies in constructing appropriate prompt words to guide the large language model to efficiently and accurately complete the task of generating the feature maps of each interface component and the style inheritance maps of each interface component based on the program code in the code repository of the front-end application. In the embodiments of the present disclosure, the above-mentioned prompt words may include the following parts: role description, task description, input data, output requirements, etc. The above-mentioned role description part is used to set the role of the large language model when performing the task. For example, in the embodiments of the present disclosure, the above-mentioned role description part may set the large language model as a front-end application developer. The above-mentioned task description part is used to set the task content currently performed by the large language model. For example, in the embodiments of the present disclosure, the above-mentioned task description part may set the large language model to parse the program code (input data) in the code repository of the front-end application to determine the feature maps of each interface component and the style inheritance maps of each interface component involved in the above-mentioned program code. The above-mentioned input data may be all the program code in the code repository of the front-end application. The above-mentioned output requirements may require the specific content and format of the feature maps of each interface component and the style inheritance maps of each interface component output by the large language model. In addition, in order to further guide the large language model to more efficiently and accurately complete the task of generating the feature maps of each interface component and the style inheritance maps of each interface component based on the program code in the code repository of the front-end application, it is also possible to further generate a thought chain related to the above-mentioned Figure 3 and Figure 4 shown in each step as a refined part of the task description in the prompt words, so as to guide the large language model to complete the task step by step according to the steps given by the thought chain. It can be seen that through the above method, appropriate prompt words can be constructed to guide the large language model to efficiently and accurately complete the task of generating the feature maps of each interface component and the style inheritance maps of each interface component based on the program code in the code repository of the front-end application.

[0089] It can be understood that the process of the above-mentioned large language model parsing the program code in the code repository of the front-end application and determining the feature maps of each interface component and the style inheritance maps of each interface component is similar to the process of fine-tuning the large language model, so that the large language model can further learn the structural features of the page components involved in the program code in the code repository of the front-end application and the inheritance relationship of styles between interface components, so as to more efficiently and accurately complete the subsequent task of locating errors in the program code.

[0090] Regarding the above step 220, in some embodiments of the present disclosure, the above page screenshot may be a screenshot of a page of a front-end application submitted by a user and containing abnormal interface components. For example, during the development, testing, or use of a front-end application, if a user discovers an abnormal user interface, the user can take a screenshot of the current page and submit the obtained page screenshot, and the front-end application error location device described in the embodiments of the present disclosure completes the location from the page screenshot to the error program code segment.

[0091] In some other embodiments of the present disclosure, the above page screenshot may also be a key image frame extracted from a screen recording video. Specifically, a screen recording video submitted by a user can be obtained first; then, at least one key image frame in the above screen recording video is extracted; finally, each of the at least one extracted key image frames is used as the above page screenshot respectively. Then, the front-end application error location method described in the embodiments of the present disclosure is executed.

[0092] Specifically, in the embodiments of the present disclosure, various methods can be used to extract key image frames from a screen recording video. These methods may include: frame difference method or dynamic threshold method and other scene mutation detection technologies, I-frame extraction method, feature clustering-based method, average value algorithm-based method, motion analysis-based method, or artificial intelligence-assisted method, etc., either alone or in combination. For example, considering the characteristics that the content of the screen recording video changes frequently locally and is mainly a static background, the I-frame extraction method can be used first to extract the I-frames (key frames) from the screen recording video to quickly obtain candidate frames and reduce redundancy; then, the block-based frame difference method and the dynamic threshold method can be used to detect local content changes in the scene (such as text editing or button clicking, etc.); and further combined with the element recognition technology on the page to further capture page state transitions (such as pop-ups or new tab pages, etc.), so as to determine the above key image frames.

[0093] It can be understood that the above solution of extracting at least one key image frame from the screen recording video and using at least one key image frame as the above page screenshot can effectively expand the application scenario of the front-end application error location method described in the embodiments of the present disclosure. That is to say, the front-end application error location method described in the embodiments of the present disclosure can not only locate the error program code segment in the front-end application based on the page screenshots submitted by the user, but also locate the error program code segment in the front-end application based on the screen recording video submitted by the user. Moreover, since the screen recording video can not only display the page anomalies caused by the style errors of the interface components in the front-end application, but also display the page anomalies caused by the state anomalies of the interface components in the front-end application, the application scenario of the front-end application error location method described in the embodiments of the present disclosure is further expanded. In this way, the front-end application error location method described in the embodiments of the present disclosure can not only effectively locate the style errors of the interface components, but also effectively locate the state errors of the interface components. It should be noted that how to locate the state errors of the interface components in the front-end application will be described in detail later and will be omitted here for the time being.

[0094] Regarding the above step 230, Figure 5 It shows the implementation process of the method for identifying the abnormal interface components in the page screenshot described in the embodiments of the present disclosure and extracting the structure information and abnormal information corresponding to the abnormal interface components. As Figure 5 shown, the specific method for identifying the abnormal interface components in the page screenshot can specifically include the following multiple steps.

[0095] In step 510, perform layout parsing on the page screenshot to extract the structure information of at least one interface component to be detected included in the page screenshot.

[0096] In an embodiment of the present disclosure, in order to detect interface components from the above page screenshot and extract the structural information of these interface components, the page screenshot can first be denoised by an image processing tool (for example, Gaussian filtering is used to eliminate the noise in the page screenshot); then, the boundaries of the interface components are strengthened by an edge detection algorithm; further, the page screenshot can be segmented based on the texture features of the image (for example, the local texture factor of the image) to complete the preliminary detection of the interface component area; and then, by combining techniques such as text area detection, optical character recognition, and image analysis, the initially detected interface components are classified, attribute-recognized, and located, so as to obtain various information such as the type, size, position, style, and relevant text of the interface components as the structural information corresponding to the above interface components to be detected. It can be understood that the structural information corresponding to the above interface components to be detected can also be represented or stored using structured data. For example, the above structured data can be data in JSON format to record various information such as the type, size, position, style, and relevant text of the interface components to be detected recognized from the page screenshot.

[0097] In step 520, abnormal interface components are identified from the at least one interface component to be detected based on the interface component library associated with the above program code and the structural information of the at least one interface component to be detected.

[0098] It can be understood that the interface component library associated with the program code of the above front-end application defines all the interface components that the above front-end application can use and their structural information. Based on the above interface component library, abnormal interface components with abnormal displays can be identified from the at least one interface component to be detected.

[0099] In step 530, the structural information and abnormal information corresponding to the abnormal interface components are determined.

[0100] In an embodiment of the present disclosure, the structural information corresponding to the abnormal interface component is the structural information corresponding to the interface component to be detected corresponding thereto, and the execution result of the above step 510 can be directly read.

[0101] In addition, in an embodiment of the present disclosure, the above abnormal information may specifically be information indicating the difference between the structural information corresponding to the above abnormal interface component and the structural information of the interface component in its corresponding component library, that is, the difference between the actual structure of the above abnormal interface component and its expected structure.

[0102] By the above method, layout analysis is performed on the page screenshot to obtain the structural information of the interface components to be detected. Combining the definition of the interface components and their structural information in the interface component library, the structural information and abnormal information corresponding to the abnormal interface components can be better identified, so as to achieve a better error localization effect.

[0103] Specifically, in some embodiments, the method for identifying abnormal interface components from the at least one interface component to be detected based on the interface component library associated with the program code and the structural information of the at least one interface component to be detected described in step 520 above may refer to Figure 6 , and includes the following multiple steps.

[0104] In step 610, at least one first target interface component corresponding to the at least one interface component to be detected is determined from the interface component library based on the structural information corresponding to the at least one interface component to be detected.

[0105] In the embodiments of the present disclosure, the corresponding first target interface component may be determined from the interface component library associated with the program code according to the type of the at least one interface component to be detected (for example, whether it is a button or a text box, etc.).

[0106] In step 620, the metadata of the at least one first target interface component is extracted from the interface component library.

[0107] In the embodiments of the present disclosure, the metadata of the above first target interface component refers to the data used to describe the first target interface component. For example, attribute information such as the name and type of the first target interface component, etc.

[0108] In step 630, the metadata of the at least one first target interface component is input into an image generation model to generate an expected page image.

[0109] In the embodiments of the present disclosure, the above image generation model may be, for example, a diffusion model or other models for image generation. The above image generation model may also be other multi-modal large models. By inputting the metadata of the at least one first target interface component into the above image generation model, the image generation ability of the image generation model can be utilized to generate a page image, and this page image should be in line with expectations, that is, it usually does not contain abnormal interface components.

[0110] In step 640, the expected page image is compared with the above page screenshot to determine the abnormal display area and the display difference information corresponding to the abnormal display area.

[0111] It can be understood that since the above page screenshot usually is a page screenshot containing abnormal interface components, there will usually be a difference between the page screenshot and the expected page image generated based on the first target interface component, and this difference can usually be considered to be caused by the abnormal interface components. Therefore, through image comparison technology, the difference between the page screenshot and the expected page image generated based on the first target interface component, that is, the abnormal display area, can be determined, and the display difference information between the two can be obtained.

[0112] In step 650, the interface component to be detected corresponding to the above abnormal display area is determined as an abnormal interface component.

[0113] After determining the abnormal interface component, in the above step 530, the structural information corresponding to the abnormal interface component can be extracted; and the display difference information corresponding to the abnormal display area is determined as the abnormal information corresponding to the abnormal interface component.

[0114] In some embodiments of the present disclosure, the process of identifying the abnormal interface component in the page screenshot in step 230 above, extracting the structural information and abnormal information corresponding to the abnormal interface component, or a certain step thereof can also be implemented using a multimodal large model. Thus, the powerful image and text processing and analysis capabilities of the multimodal large model can also be fully utilized to quickly and accurately identify the abnormal interface component in the page screenshot and extract the structural information and abnormal information corresponding to the abnormal interface component.

[0115] In a specific application, similar to the foregoing steps, the key point of using the multimodal large model to execute step 230 above or a specific step thereof also lies in constructing appropriate prompt words to guide the multimodal large model to efficiently and accurately complete the relevant tasks. Taking the task of identifying the abnormal interface component in the page screenshot and extracting the structural information and abnormal information corresponding to the abnormal interface component as an example, in the embodiments of the present disclosure, the above prompt words may include the following parts: role description, task description, input data, and output requirements. The above role description part is used to set the role of the multimodal large model when performing the task. For example, in the embodiments of the present disclosure, the above role description part can set the multimodal large model as a front-end application developer. The above task description part is used to set the task content currently executed by the multimodal large model. For example, in the embodiments of the present disclosure, the above task description part can set the multimodal large model to identify the abnormal interface component in the page screenshot and extract the structural information and abnormal information corresponding to the abnormal interface component. The above input data can be the above page screenshot. The above input data can also include: the component library associated with the program code of the above front-end application. The above output requirements can require the format of the structural information and abnormal information of the abnormal interface component output by the multimodal large model. In addition, in order to further guide the multimodal large model to more efficiently and accurately complete the above tasks, it can also be further based on the above Figure 5 and Figure 6Each of the steps shown above generates a thought chain related to the task of generating abnormal interface components in the above-mentioned recognized page screenshot and extracting the corresponding structure information and abnormal information of the abnormal interface components, as a refined part of the task description in the prompt, so as to guide the multi-modal large model to gradually complete the relevant tasks according to the steps given by the thought chain. It can be seen that through the above method, appropriate prompt words can be constructed to guide the multi-modal large model to efficiently and accurately complete the task of recognizing abnormal interface components in the above-mentioned page screenshot and extracting the corresponding abnormal information of the abnormal interface components.

[0116] Furthermore, when the user submits the page screenshot, the user can further submit a description of the abnormal interface component associated with the page screenshot. For example, when the user finds that the display of button A on the current page is abnormal, the user can submit a screenshot of the current page and enter an abnormal interface component description such as "The display of button A is abnormal". In this case, the above-mentioned abnormal interface component description will be used as Figure 1 input data for the multi-modal large model, so as to further prompt the multi-modal large model of the abnormal situation of the abnormal interface component, to assist the multi-modal large model to more efficiently and accurately complete the task of recognizing abnormal interface components in the above-mentioned page screenshot and extracting the corresponding abnormal information of the abnormal interface components.

[0117] Regarding step 240, the specific method of locating the error program code segment corresponding to the above-mentioned abnormal interface component from the above-mentioned program code according to the structure information and abnormal information corresponding to the above-mentioned abnormal interface component, the feature map of each interface component, and the style inheritance map of each interface component can be as Figure 7 shown, including the following multiple steps:

[0118] In step 710, match the structure information corresponding to the abnormal interface component with the feature maps of each interface component to determine the second target interface component corresponding to the abnormal interface component.

[0119] Specifically, in the embodiment of the present disclosure, the structure information corresponding to the above-mentioned abnormal interface component can be matched with the attribute patterns in the feature maps of each interface component, so as to determine the second target interface component corresponding to the abnormal interface component.

[0120] In step 720, locate the target program code corresponding to the second target interface component in the above-mentioned program code.

[0121] It can be understood that after determining the second target interface component corresponding to the abnormal interface component, the target program code corresponding to the second target interface component can be located in the code repository of the front-end application program.

[0122] It can be seen that in the above embodiments of the present disclosure, by matching the structural information corresponding to the abnormal interface component with the characteristic maps of each interface component determined in advance, it is possible to determine which actual second target interface component corresponds to the abnormal interface component, and according to the characteristic map, it is also possible to quickly map to the target program code part corresponding to the second target interface component in the program code. Thus, it can be seen that through the characteristic maps of each interface component established in advance, the association between vision and code can be quickly and accurately completed, that is, the quick mapping from the abnormal interface component on the page to the target program representative part can be quickly and accurately completed.

[0123] In step 730, based on the abnormal information corresponding to the above abnormal interface component, the characteristic maps of each interface component, and the style inheritance maps of each interface component, locate the error program code segment corresponding to the above abnormal information in the target program code.

[0124] Next, it is also possible to further locate the error program code segment that causes the interface component abnormality from the target program code part based on the abnormal information corresponding to the abnormal interface component. It can be understood that since the above abnormal information represents the difference between the abnormal display area on the page screenshot and the expected page image, therefore, according to this difference, the error program code segment that causes the interface component abnormality can be directly located. Even when an abnormality occurs in the style inheritance of the interface component, the relevant attributes of the parent interface component where the abnormality occurs can be directly located through the style inheritance maps of each interface component. As mentioned above, through the characteristic maps of each interface component established in advance, the association between vision and code can be quickly and accurately completed, that is, the quick mapping from the abnormal interface component on the page to the target program representative part can be quickly and accurately completed.

[0125] In addition, since the style inheritance maps of each interface component describe the inheritance relationship of the interface component styles and reflect the hierarchical relationship between the interface components, therefore, through the front-end application program error location method described in the embodiments of the present disclosure, the program code errors of the upper-level interface components can be traced, so that the error location is more accurate and effective.

[0126] In some embodiments of the present disclosure, the above step 240 can also be implemented by a large language model. Optionally, the same large language model used in the above step 210 can be used. The implementation solution for locating the error program code segment described in the above step 240 by the large language model can make full use of the advantages of the large language model in semantic understanding, and can effectively utilize the feature maps of each interface component and the style inheritance maps of each interface component established in the previous link, so as to efficiently and accurately determine the target program code from the code repository of the front-end application program, and further locate the error program code segment from the target program code, thereby improving the effectiveness and accuracy of the above front-end application program error location method, and at the same time greatly reducing the development cost of the above front-end application program error location solution.

[0127] Similar to the foregoing embodiments, in a specific application, the key point of using the large language model to execute the above step 240 is also to construct appropriate prompt words to guide the large language model to efficiently and accurately complete the task of locating the error program code segment. In the embodiments of the present disclosure, the above prompt words may include the following parts: role description, task description, input data, and output requirements. The above role description part is used to set the role of the large language model when performing tasks. For example, in the embodiments of the present disclosure, the above role description part can set the large language model as a front-end application developer. The above task description part is used to set the task content currently executed by the large language model. For example, in the embodiments of the present disclosure, the above task description part can set the large language model to determine the error program code segment corresponding to the abnormal interface component from the program code according to the structure information and abnormal information (input information) corresponding to the abnormal interface component, the feature maps of each interface component, and the style inheritance maps of each interface component. Since all the program codes in the code repository of the front-end application program are the inputs of the large language model in the previous link, and the feature maps of each interface component and the style inheritance maps of each interface component are the outputs of the large language model in the previous link, that is, the learned knowledge. Therefore, in the current task, there is no need to repeatedly input the above data to the large language model. The above output requirements can require the content and format of the error program code segment output by the large language model. In addition, in order to further guide the large language model to more efficiently and accurately complete the task of locating the error program code segment, a thought chain related to the task of locating the error program code segment can be further generated based on each step shown in the above Figure 7 As shown, as a refined part of the task description in the prompt words, the large language model can complete the relevant tasks step by step according to the steps given by the thought chain. It can be seen that through the above method, appropriate prompt words can be constructed to guide the large language model to efficiently and accurately complete the task of locating the error program code segment.

[0128] It can be seen from this that through the above front-end application error localization method, abnormal interface components can be identified based on the page screenshots or screen recording videos uploaded by users, and further, the error program code segments associated with the abnormal interface components can be located according to the feature maps of each interface component established based on all the program codes of the front-end application and the style inheritance maps of each interface component. By using the established feature maps of each interface component and the style inheritance maps of each interface component, the mapping from vision to code is realized, effectively solving the problem that the existing front-end application error detection tools rely on users to manually associate page anomalies with code logic and cannot automatically map the user interface anomalies in the user interface screenshots / screen recording videos submitted by users to specific program codes.

[0129] Furthermore, as mentioned above, since the style inheritance maps of the above-mentioned interface components describe the inheritance relationship of the interface component styles and reflect the hierarchical relationship between the interface components, therefore, through the front-end application error localization method described in the embodiments of the present disclosure, the program code errors of the superior interface components can be traced, thus effectively solving the problem that the existing front-end application error detection tools are limited to line-level error localization and cannot understand the hierarchical relationship between interface components, resulting in fragmented program repair solutions.

[0130] It can be understood that the above front-end application error localization method can mainly be used to locate the front-end application code that causes display anomalies of interface components. For example, style errors of interface components, etc. In addition to the above anomalies, in the front-end application, there are also page anomalies caused by incorrect state transmission of interface components. For example, after pressing a certain button, the page should jump but does not, or when a user's operation triggers an event that should change the state of an interface component, but the state of the interface component actually does not change, etc. For these page anomalies caused by incorrect state transmission of interface components, it is usually necessary to analyze the screen recording video to determine the cause of the page anomaly and locate the error program code in the front-end application.

[0131] To solve the above problems, after extracting at least one key image frame from the screen recording video submitted by the user or while extracting the above key image frames, in addition to using the above key image frames as the page screenshots to execute the above front-end application error localization method, the embodiments of the present disclosure further provide a method for locating the state transmission error of interface components in the front-end application.

[0132] Figure 8 Shows the implementation process of the method for locating the state transmission error of interface components in the front-end application described in the embodiments of the present disclosure. As Figure 8As shown, after extracting the key image frames from the screen recording video submitted by the user, the following multiple steps of operations will be further performed.

[0133] In step 810, determine the user operation sequence corresponding to the screen recording video.

[0134] Specifically, in the embodiments of the present disclosure, the operations of the user can be captured from the screen recording video or at least one key image frame by combining various traditional image processing methods including timeline segmentation, optical character recognition, dynamic event capture, etc. In addition, if the quality of the screen recording video is high, the finger movement trajectory can also be tracked by the optical flow method. For example, the interface components included in the key image frame can be determined by the method of identifying interface components from the page screenshot mentioned above. Then, by detecting the changes in the interface components or the page between the front and back key image frames, the type of user operation and the object being operated, that is, the interface component on the page being operated, can be determined. For example, by detecting that the state of button A has changed between the front and back key image frames, it can be determined that the user has performed a click operation, and the object being operated is button A. Another example is that by detecting that the text content in input box B has changed between the front and back key image frames, it can be determined that the user has performed a text input operation, and the object being operated is input box B. Further, the specific text content input by the user can also be determined by optical character recognition. That is to say, in some embodiments of the present disclosure, the above user operation sequence will include at least one operation interface component arranged in chronological order. Each interface component operation can include: operation type (for example, click, text input, swipe, etc.) and the object being operated (for example, button, input box, etc. on the page). In addition, each interface component operation can also include: operation time (for example, timestamp), operation position (for example, coordinates on the page), and the content of the operation (for example, the input text), etc. It can be understood that the above user operation sequence can be represented or stored using structured data. For example, it can be represented or stored in JSON format.

[0135] It should be noted that for the screen recording video, the key image frames usually correspond to the user operations. For example, the front and back two key image frames respectively represent the pages before and after the user performs a certain operation. Therefore, in actual applications, the step of determining the user operation sequence corresponding to the above screen recording video based on the above key image frames can be completed together with the operation of extracting key image frames from the screen recording video. For example, when determining the key image frames based on the scene change detection technology, the operation type, operation time, operation position, and the object being operated of the interface component operation performed by the user can be directly determined, so as to directly determine the user operation sequence.

[0136] In step 820, based on the user operation sequence and the feature maps of each interface component, determine the third target interface component corresponding to the object being operated and the abnormal state of the third target interface component.

[0137] In the embodiments of the present disclosure, based on the object being operated corresponding to the user operation sequence, the third target interface component corresponding to the object being operated can be determined by matching with the feature maps of each interface component. In addition, based on the operation type in the operation sequence and the feature map of the third target interface component, determine the state of each interface component operation in the user operation sequence corresponding to the third target interface component. Finally, determine the abnormal state of the third target interface component by determining whether the state of each interface component operation in the user operation sequence corresponding to the third target interface component meets the expectation.

[0138] Specifically, in some embodiments of the present disclosure, the above step 820 may specifically include the following multiple steps.

[0139] First, convert the above user operation sequence into an event stream.

[0140] In the embodiments of the present disclosure, the above event stream generally refers to a sequence of events arranged in chronological order. The above event sequence includes at least one event (for example, click, input, scrolling, etc.). Wherein, each event represents an operation of the user. Each event may also include: the type of the event, the time when the event occurs, and the object associated with the event (i.e., the object being operated), etc.

[0141] Secondly, match the above event stream with the event binding information of the object being operated in the feature maps of each interface component to determine the third target interface component corresponding to the object being operated, and restore the event processing logic triggered by the user operation.

[0142] In an embodiment of the present disclosure, after determining the event stream, for each event in the above event stream, the object associated with the event (i.e., the above-mentioned object to be operated) can be first matched with the feature maps of each pre-established interface component to determine the interface component corresponding to the object associated with the event in the front-end application, that is, the above-mentioned third target interface component. Then, further match the event binding information in the feature map of the third target interface component with the type of the event to restore the event handling logic (e.g., event handling function) triggered by the user operation. For example, event X is a click event, and the object associated with the event is button A. Then, in the above step 730, first, determine the third target interface component S corresponding to button A in the front-end application, and based on the event binding information of the third target interface component S, determine the processing function F of the click event corresponding to the target interface component S. That is, when button A is clicked, the front-end application will execute the processing function F of the click event in the program code corresponding to the third target interface component S.

[0143] Next, determine the state transfer path of the third target interface component based on the above event handling logic and the state dependency information of the third target interface component in the feature maps of each interface component.

[0144] Specifically, after determining the event handling logic, based on the state dependency information in the feature map of the above-mentioned third target interface component, the state of the third target interface component after executing the event handling logic can be further determined. As mentioned above, in the above step, it is determined that the processing function F corresponding to event X is a click event. In this way, it can be determined that the page will jump after the processing function F of the click event is executed. In this way, the state transfer path of the third target interface component S, that is, the actual state transfer path, can be obtained.

[0145] Finally, determine the abnormal state of the third target interface component based on the state transfer path of the third target interface component.

[0146] In an embodiment of the present disclosure, the state transfer path of the above-mentioned third target interface component can be checked. For example, the state transfer path of the above-mentioned third target interface component can be compared with the expected state transfer path of the above-mentioned third target interface component to determine the abnormal state of the third target interface component. For example, after clicking on a certain target interface component, the page should jump, but actually the state transfer path of the target interface component is that the color changes from a to b, which indicates that this target interface component has an abnormal state, resulting in an abnormal page.

[0147] It should be noted that the expected state transfer path of the above-mentioned third target interface component can be a pre-set state transfer path, such as the state transfer path set during the development of the front-end application. In addition, the expected state transfer path of the above-mentioned third target interface component can also be a state transfer path determined according to the exception description submitted by the user while submitting the screen recording video. For example, when the user clicks button A and no page jump occurs, the user can submit a screen recording video related to the above operation and at the same time submit an exception description of "no page jump occurs when clicking button A". Based on the description submitted by the user, the expected state transfer path can be determined as that the page should jump after the processing function F of button A is executed by clicking.

[0148] After determining the abnormal state of the third target interface component, step 830 below will continue to be executed.

[0149] In step 830, based on the abnormal state of the third target interface component and the feature maps of each interface component, the above-mentioned error program code segment is determined from the above program code.

[0150] Specifically, after determining the third target interface component, the target program code corresponding to the third target interface component can be determined. Next, in the above step 830, the error program code segment can be located from the target program code based on the abnormal state of the above-mentioned third target interface component, its state dependency information, and event binding information.

[0151] In the embodiments of the present disclosure, the process of converting the screen recording video submitted by the user into a user operation sequence and extracting key image frames can be implemented using a multimodal large model. Specifically, the above-mentioned multimodal large model can use the multimodal large model described in the foregoing embodiments. After obtaining the user operation sequence, the large language model in the foregoing embodiments can also be used to perform the task of locating the error program code based on the user operation sequence. The method of using the multimodal large model and the large language model to cooperate to realize the error location of the front-end application program code can make full use of the powerful image and text processing and analysis capabilities of the multimodal large model and the text processing capabilities of the large language model, and can quickly and accurately complete the conversion task from the screen recording video to the user operation sequence, the extraction of key image frames, and the task of locating the error program code based on the user operation sequence.

[0152] In a specific application, similar to the foregoing steps, the key to using the multimodal large model to convert the screen recording video submitted by the user into a user operation sequence and extract key image frames also lies in constructing appropriate prompt words to guide the multimodal large model to efficiently and accurately complete the above tasks. In the embodiments of the present disclosure, the above prompt words may also include the following parts: role description, task description, input data, and output requirements. The above role description part is used to set the role of the multimodal large model when performing tasks. For example, in the embodiments of the present disclosure, the above role description part may set the multimodal large model as a front-end application developer. The above task description part is used to set the task content currently executed by the multimodal large model. For example, in the embodiments of the present disclosure, the above task description part may set the multimodal large model to convert the screen recording video submitted by the user into a user operation sequence and extract the key image frames therein. The above input data may be the above screen recording video. The above output requirements may require the formats of the key image frames and event streams output by the multimodal large model. In addition, in order to further guide the multimodal large model to more efficiently and accurately complete the above tasks, a thought chain related to the task of converting the screen recording video submitted by the user into a user operation sequence and extracting the key image frames therein may be further generated based on the above key image frame extraction method and the method of step 810 as a refined part of the task description in the prompt words, so as to guide the multimodal large model to gradually complete the relevant tasks according to the steps given by the thought chain. It can be seen that through the above method, appropriate prompt words can be constructed to guide the multimodal large model to efficiently and accurately complete the task of converting the screen recording video submitted by the user into a user operation sequence and extracting the key image frames therein.

[0153] In addition, in specific applications, the key to using a large language model to perform the above task of locating incorrect program code based on the user operation sequence also lies in constructing appropriate prompt words to guide the large language model to efficiently and accurately complete the above task. In the embodiments of the present disclosure, the above prompt words may also include the following parts: role description, task description, input data, and output requirements. The above role description part is used to set the role of the large language model when performing the task. For example, in the embodiments of the present disclosure, the above role description part may set the large language model as a front-end application developer. The above task description part is used to set the task content currently performed by the large language model. For example, in the embodiments of the present disclosure, the above task description part may set the large language model to locate incorrect program code based on the user operation sequence. Since all the program code in the code repository of the front-end application is the input of the large language model in the previous link, and the feature maps of each interface component and the style inheritance maps of each interface component are the output of the large language model in the previous link, that is, the knowledge that has been learned. Therefore, in the current task, there is no need to repeat the input of the above data to the large language model. The above output requirements can require the content and format of the incorrect program code segment output by the large language model. In addition, in order to further guide the large language model to more efficiently and accurately complete the task of locating the incorrect program code segment, a thought chain related to the task of locating the incorrect program code segment can be further generated based on the above steps 820-820 as a refinement part of the task description in the prompt words, so as to guide the large language model to gradually complete the relevant tasks according to the steps given by the thought chain. It can be seen that through the above method, appropriate prompt words can be constructed to guide the large language model to efficiently and accurately complete the task of locating the incorrect program code segment.

[0154] Furthermore, when the user submits the screen recording video, the user can further submit an abnormal interface component description associated with the screen recording video. For example, when the user finds that the page should jump after button A on the current page is clicked but does not jump, the user can submit a screen recording video of the above operation process and input an abnormal interface component description such as "the page does not jump after button A is clicked". In this case, the above abnormal interface component description and the user operation sequence related to the screen recording video will be used as the input data of the large language model, so as to further prompt the abnormal state of the abnormal interface component to the large language model to assist the large language model to more efficiently and accurately complete the task of locating the incorrect program code segment.

[0155] Thus, based on Figure 8The method shown can achieve error localization for page exceptions caused by abnormal states of interface components in a front-end application. That is, when there are abnormal states of interface components in the program code of the front-end application (for example, there are exceptions in the declaration or use of states, event handling functions have exceptions, etc.), the complete reproduction of the user operation chain can be achieved through the analysis of multiple screen recording videos, and based on the error states of the interface components that appear in the reproduced user operation sequence, the error program code in the front-end application can be located based on the pre-established feature maps of each interface component and the style inheritance maps of each interface component, thus effectively achieving the goal of error localization in the front-end application.

[0156] Corresponding to the above front-end application error localization method, some embodiments of the present disclosure also disclose a front-end application error localization device. Figure 9 Shows the internal structure of the front-end application error localization device described in the embodiments of the present disclosure. As Figure 9 shown, the above front-end application error localization device may include the following multiple modules:

[0157] A map generation module 910, configured to parse the program code in the code repository of the front-end application to determine the feature maps of each interface component involved in the program code and the style inheritance maps of each interface component;

[0158] An image receiving module 920, configured to receive a page screenshot corresponding to the front-end application;

[0159] An abnormal interface component recognition module 930, configured to recognize the abnormal interface components in the page screenshot, and extract the structure information and abnormal information corresponding to the abnormal interface components; and

[0160] A localization module 940, configured to locate the error program code segment corresponding to the abnormal interface component from the program code according to the structure information and abnormal information corresponding to the abnormal interface component, the feature maps of each interface component, and the style inheritance maps of each interface component.

[0161] It should be noted that the implementation methods of each module in the above device and the specific technical effects that can be achieved can refer to the implementation methods of each step in the foregoing embodiments, and will not be repeated here.

[0162] Based on the same inventive concept, corresponding to the method of any of the above embodiments, the present disclosure also provides an electronic device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor implements the front-end application error localization method described in any of the above embodiments when executing the computer program.

[0163] Figure 10 FIG. 1 shows a more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment. The device may include: a processor 2010, a memory 2020, an input / output interface 2030, a communication interface 2040, and a bus 2050. Among them, the processor 2010, the memory 2020, the input / output interface 2030, and the communication interface 2040 are communicatively connected to each other inside the device through the bus 2050.

[0164] The processor 2010 may be implemented in a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0165] The memory 2020 may be implemented in the form of a ROM (Read Only Memory), a RAM (Random Access Memory), a static storage device, a dynamic storage device, etc. The memory 2020 may store an operating system and other application programs. When implementing the technical solutions provided in the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 2020 and are called and executed by the processor 2010.

[0166] The input / output interface 2030 is used to connect input / output devices to achieve information input and output. Among them, the input / output devices may be configured as components in the device or externally connected to the device to provide corresponding functions. The input devices may include microphones, various sensors, etc., and the output devices may include displays, speakers, vibrators, indicator lights, etc.

[0167] The communication interface 2040 is used to connect a communication module (not shown in the figure) to achieve communication and interaction between this device and other devices. The communication module may communicate through a wired method (such as USB, network cable, etc.) or through a wireless method (such as mobile network, WIFI, Bluetooth, etc.).

[0168] The bus 2050 includes a path for transmitting information between various components of the device (such as the processor 2010, the memory 2020, the input / output interface 2030, and the communication interface 2040).

[0169] It should be noted that although the above device only shows the processor 2010, the memory 2020, the input / output interface 2030, the communication interface 2040, and the bus 2050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, those skilled in the art can understand that the above device may also only include the components necessary to implement the solution of the embodiments of this specification, and does not necessarily include all the components shown in the figure.

[0170] The electronic device of the above embodiment is used to implement the corresponding front-end application program error location method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be elaborated here.

[0171] Based on the same inventive concept, corresponding to the method of any of the above embodiments, the present disclosure also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the front-end application program error location method as described in any of the foregoing embodiments.

[0172] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tapes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device.

[0173] The computer instructions stored in the storage medium of the above embodiment are used to cause the computer to execute the front-end application program error location method as described in any of the foregoing embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be elaborated here.

[0174] Based on the same inventive concept, corresponding to the front-end application program error localization method of any of the above embodiments, the present disclosure also provides a computer program product, which includes computer program instructions. In some embodiments, when the computer program instructions run on a computer, the computer is caused to execute the steps in each of the embodiments of the front-end application program error localization method. Corresponding to the execution subjects of the steps in each of the embodiments of the front-end application program error localization method, the processor that executes the corresponding steps may belong to the corresponding execution subject.

[0175] The computer program product of the above embodiments is used to cause a processor to execute the front-end application program error localization method described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be elaborated here.

[0176] Those of ordinary skill in the art should understand that: the discussion of any of the above embodiments is only exemplary and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples; under the idea of the present disclosure, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations in different aspects of the embodiments of the present disclosure as described above, which are not provided in detail for the sake of brevity.

[0177] In addition, for the sake of simplicity of description and discussion, and in order not to make the embodiments of the present disclosure difficult to understand, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. In addition, the device may be shown in block diagram form to avoid making the embodiments of the present disclosure difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure are to be implemented (i.e., these details should be completely within the understanding of those skilled in the art). In the case where specific details (such as circuits) are set forth to describe the exemplary embodiments of the present disclosure, it will be apparent to those skilled in the art that the embodiments of the present disclosure may be implemented without these specific details or with variations of these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0178] Although the present disclosure has been described in conjunction with specific embodiments of the present disclosure, many alternatives, modifications, and variations of these embodiments will be apparent to those of ordinary skill in the art based on the foregoing description. For example, other memory architectures (such as dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0179] Embodiments of the present disclosure are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure shall be included within the protection scope of the present disclosure.

Claims

1. A method for locating front-end application errors, comprising: Parsing the program code in the code repository of the front-end application to determine the feature maps of each interface component involved in the program code and the style inheritance maps of each interface component; Receiving a page screenshot corresponding to the front-end application; Identifying abnormal interface components in the page screenshot, and extracting the structural information and abnormal information corresponding to the abnormal interface components; And Locating the error program code segment corresponding to the abnormal interface component from the program code according to the structural information and abnormal information corresponding to the abnormal interface component, the feature maps of each interface component, and the style inheritance maps of each interface component.

2. The method according to claim 1, wherein The feature maps of each interface component include: the identifiers of each interface component, the attribute modes of each interface component, the state dependency information of each interface component, and the event binding information of each interface component; and the style inheritance maps of each interface component include: the inheritance relationships between the styles of each interface component.

3. The method according to claim 2, wherein The parsing of the program code in the code repository of the front-end application to determine the feature maps of each interface component involved in the program code and the style inheritance maps of each interface component includes: Parsing the script program code in the code repository of the front-end application to determine the feature maps of each interface component; and Parsing the component style program code in the code repository of the front-end application to determine the style inheritance maps of each interface component.

4. The method according to claim 3, wherein, The parsing of the script program code in the code repository of the front-end application to determine the feature maps of each interface component includes: Converting the script program code into an abstract syntax tree; Identifying at least one component definition node in the abstract syntax tree; For each component definition node in the at least one component definition node respectively, Extracting the component name of the component definition node as the identifier of an interface component; Determining the attribute mode of the interface component based on the metadata of the attribute parameters associated with the component definition node; Determining the state dependency information of the interface component based on the declaration and use of the state variables associated with the component definition node; Determining the event binding information of the interface component based on the event handling logic associated with the component definition node; and Generating the feature map of the interface component based on the identifier of the interface component, the attribute mode of the interface component, the state dependency information of the interface component, and the event binding information of the interface component.

5. The method according to claim 3, wherein, The parsing of the component style program code in the code repository of the front-end application to determine the style inheritance maps of each interface component includes: Determining the hierarchical relationship between each interface component based on the nesting relationship of the component style program code; For each of the interface components, based on the component style program code, extract the style attributes of the interface component. In response to determining that the interface component has set inheritable style attributes and has not overridden the inheritable style attributes, determine the parent component of the interface component based on the hierarchical relationship, and determine that there is an inheritance relationship corresponding to the inheritable style attributes from the parent component to the interface component; and Based on the inheritance relationship corresponding to the inheritable style attributes, generate a style inheritance graph of the interface components.

6. The method according to claim 1, wherein Identifying the abnormal interface components in the page screenshot and extracting the corresponding structure information and abnormal information of the abnormal interface components includes:[[]] Perform layout parsing on the page screenshot and extract the structure information corresponding to at least one interface component to be detected included in the page screenshot; Based on the interface component library associated with the program code and the structure information of the at least one interface component to be detected, identify the abnormal interface components from the at least one interface component to be detected; and Determine the structure information and abnormal information corresponding to the abnormal interface components.

7. The method according to claim 6, wherein, The identifying the abnormal interface components from the at least one interface component to be detected based on the interface component library associated with the program code and the structure information of the at least one interface component to be detected includes:[[]] Based on the structure information corresponding to the at least one interface component to be detected, determine at least one first target interface component corresponding to the at least one interface component to be detected from the interface component library; Extract the metadata of the at least one first target interface component from the interface component library; Input the metadata of the at least one first target interface component into an image generation model to generate an expected page image; Compare the expected page image with the page screenshot to determine the abnormal display area and the display difference information corresponding to the abnormal display area; and Determine the interface component to be detected corresponding to the abnormal display area as the abnormal interface component; and The determining the structure information and abnormal information corresponding to the abnormal interface components includes:[[]] Extract the structure information corresponding to the abnormal interface component; and Determine the display difference information corresponding to the abnormal display area as the abnormal information corresponding to the abnormal interface component.

8. The method according to claim 2, wherein The locating the error program code segment corresponding to the abnormal interface component from the program code according to the structure information and abnormal information corresponding to the abnormal interface component, the feature graphs of the interface components, and the style inheritance graphs of the interface components includes:[[]] Match the structure information corresponding to the abnormal interface component with the feature graphs of the interface components to determine a second target interface component corresponding to the abnormal interface component; Locate the target program code corresponding to the second target interface component in the program code; and Locate the error program code segment corresponding to the exception information in the target program code based on the exception information corresponding to the abnormal interface component, the feature maps of each interface component, and the style inheritance maps of each interface component.

9. The method according to claim 1, wherein The parsing of the program code in the code repository of the front-end application to determine the feature maps of each interface component and the style inheritance maps of each interface component involved in the program code includes: parsing the program code based on a large language model to determine the feature maps of each interface component and the style inheritance maps of each interface component involved in the program code; The identifying the abnormal interface component in the page screenshot and extracting the structure information and exception information corresponding to the abnormal interface component includes: identifying the abnormal interface component in the page screenshot based on a multimodal large model and extracting the structure information and exception information corresponding to the abnormal interface component; and The locating the error program code segment corresponding to the abnormal interface component from the program code according to the structure information and exception information corresponding to the abnormal interface component, the feature maps of each interface component, and the style inheritance maps of each interface component includes: locating the error program code segment corresponding to the abnormal interface component from the program code based on the large language model according to the structure information and exception information corresponding to the abnormal interface component, the feature maps of each interface component, and the style inheritance maps of each interface component.

10. The method according to claim 2, wherein, The receiving the page screenshot corresponding to the front-end application includes: Receiving the screen recording video corresponding to the front-end application; Extracting at least one key image frame from the screen recording video; and Using each of the key image frames as the page screenshot.

11. The method according to claim 10, further comprising: Determining a user operation sequence corresponding to the screen recording video; wherein the user operation sequence includes at least one interface component operation; the interface component operation includes: an operation type and an object to be operated; Determining a third target interface component corresponding to the object to be operated and an abnormal state of the third target interface component based on the user operation sequence and the feature maps of each interface component; and Determining the error program code segment from the program code based on the abnormal state of the third target interface component and the feature maps of each interface component.

12. The method according to claim 11, wherein, The determining the third target interface component corresponding to the object to be operated and the abnormal state of the third target interface component based on the user operation sequence and the feature maps of each interface component includes: Converting the user operation sequence into an event stream; Matching the event stream with the event binding information of the object to be operated in the feature maps of each interface component to determine the third target interface component corresponding to the object to be operated and restore the event handling logic triggered by the user operation; Determine the state transfer path of the third target interface component based on the event processing logic and the state dependency information of the third target interface component in the feature maps of the respective interface components; and Determine the abnormal state of the third target interface component based on the state transfer path of the third target interface component.

13. A front-end application error localization device, comprising: A map generation module, configured to parse program code in a code repository of a front-end application, and determine feature maps of respective interface components involved in the program code and style inheritance maps of the respective interface components; An image receiving module, configured to receive a page screenshot corresponding to the front-end application; An abnormal interface component identification module, configured to identify an abnormal interface component in the page screenshot, and extract structure information and abnormal information corresponding to the abnormal interface component; And A localization module, configured to locate an error program code segment corresponding to the abnormal interface component from the program code according to the structure information and abnormal information corresponding to the abnormal interface component, the feature maps of the respective interface components, and the style inheritance maps of the respective interface components.

14. An electronic device, comprising: A memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the computer program, the front-end application error localization method according to any one of claims 1-12 is implemented.

15. A non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the front-end application error localization method according to any one of claims 1-12.

16. A computer program product comprising computer program instructions which, when run on a computer, cause the computer to execute the front-end application error localization method according to any one of claims 1-12.

Images