Information processing method and device and electronic equipment

Abstract

The embodiment of the invention discloses an information processing method and device and electronic equipment. In the method, after a corresponding first operation sequence is obtained through a first operation process executed on a target program, a second operation process can be executed on the target program through the first operation sequence, interface information and a neural network model, and the second operation process comprises operation executed on the target program based on a second operation sequence. The second operation sequence is obtained by the neural network model based on the interface information and the first operation sequence, and the second operation process is detected, so that the detection efficiency and comprehensiveness are improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more particularly, to an information processing method, apparatus, and electronic device. Background Art

[0002] With the development of technology, the functions of electronic devices have become increasingly rich. Among them, by detecting an electronic device, problems existing in the electronic device can be found. For example, by detecting an application program, problems existing in the application program can be found so that the application program can be improved. However, the efficiency of related detection methods still needs to be improved. Summary of the Invention

[0003] This application provides an information processing method, apparatus, and electronic device to improve the above problems.

[0004] In a first aspect, this application provides an information processing method, the method including: obtaining a corresponding first operation sequence through a first operation process executed by a target program; performing a second operation process on the target program through the first operation sequence, interface information, and a neural network model, where the second operation process includes performing an operation on the target program based on a second operation sequence, and the second operation sequence is obtained by the neural network model based on the interface information and the first operation sequence, and the interface information includes information of the interface corresponding to the first operation sequence; detecting the second operation process.

[0005] In a second aspect, this application provides an information processing apparatus, the apparatus including: an operation sequence obtaining unit, configured to obtain a corresponding first operation sequence through a first operation process executed by a target program; an operation execution unit, configured to perform a second operation process on the target program through the first operation sequence, interface information, and a neural network model, where the second operation process includes performing an operation on the target program based on a second operation sequence, and the second operation sequence is obtained by the neural network model based on the interface information and the first operation sequence, and the interface information includes information of the interface corresponding to the first operation sequence; a detection unit, configured to detect the second operation process.

[0006] In a third aspect, this application provides an electronic device, including a processor and a memory; one or more programs are stored in the memory and configured to be executed by the processor to implement the above method.

[0007] In a fourth aspect, this application provides a computer-readable storage medium, in which program code is stored, where the above method is executed when the program code is run by a start controller.

[0008] Fifth aspect, the present application provides a computer program product, which stores at least one program code for being executed by a processor to implement the above method.

[0009] An information processing method, apparatus and electronic device provided by the present application. In this method, after obtaining a corresponding first operation sequence through a first operation process performed on a target program, a second operation process can be performed on the target program through the first operation sequence, interface information and a neural network model. Among them, the second operation process includes performing an operation on the target program based on a second operation sequence, and the second operation sequence is obtained by the neural network model based on the interface information and the first operation sequence, and the second operation process is detected. Thus, through the above method, after obtaining the first operation sequence corresponding to the target program, the neural network model can perform expansion of the operation sequence based on the interface information corresponding to the first operation sequence and the first operation sequence to generate more operation sequences (the second operation sequence), so that the target program can be operated based on the generated operation sequence to achieve a more comprehensive detection of the target program. Among them, since the second operation sequence is automatically expanded by the neural network model, it is beneficial to improve the detection efficiency. Description of the Drawings

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present application. For those skilled in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0011] Figure 1 A schematic diagram showing an application scenario of the information processing method proposed in the embodiment of the present application;

[0012] Figure 2 A schematic diagram showing another application scenario of the information processing method proposed in the embodiment of the present application;

[0013] Figure 3 A schematic diagram showing still another application scenario of the information processing method proposed in the embodiment of the present application;

[0014] Figure 4 A flowchart showing an information processing method proposed in the embodiment of the present application;

[0015] Figure 5 A schematic diagram showing the recognition of an interface by the target information processing method in the embodiment of the present application;

[0016] Figure 6Shows a schematic diagram of the interface content feature sequence in an embodiment of the present application;

[0017] Figure 7 Shows a flowchart of an information processing method proposed in another embodiment of the present application;

[0018] Figure 8 Shows a schematic diagram of pixels at the same position in an embodiment of the present application;

[0019] Figure 9 Shows a schematic diagram of obtaining a first operation sequence in an embodiment of the present application;

[0020] Figure 10 Shows a flowchart of an information processing method proposed in yet another embodiment of the present application;

[0021] Figure 11 Shows a timing diagram of an information processing method proposed in an embodiment of the present application;

[0022] Figure 12 Shows a structural block diagram of an information processing device proposed in an embodiment of the present application;

[0023] Figure 13 Shows a structural block diagram of an electronic device for executing the information processing method according to an embodiment of the present application;

[0024] Figure 14 Is a storage unit for storing or carrying program code for implementing the information processing method according to an embodiment of the present application. Detailed implementation manners

[0025] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0026] With the continuous progress of technology, the functions of electronic devices are becoming increasingly diverse. In this process, various potential problems of electronic devices can be accurately located by means of detection means. Take application programs as an example. When detecting them, their deficiencies can be identified, providing a direction for subsequent optimization and improvement work. However, the related information processing methods still need to be improved in terms of efficiency.

[0027] Therefore, the inventor proposes an information processing method. In this method, after obtaining a corresponding first operation sequence through the first operation process performed on the target program, the second operation process can be performed on the target program through the first operation sequence, interface information, and a neural network model. Among them, the second operation process includes performing an operation on the target program based on a second operation sequence, and the second operation sequence is obtained by the neural network model based on the interface information and the first operation sequence, and the second operation process is detected. Thus, through the above method, after obtaining the first operation sequence corresponding to the target program, the neural network model can perform the expansion of the operation sequence based on the interface information corresponding to the first operation sequence and the first operation sequence to generate more operation sequences (the second operation sequence), so that the target program can be operated based on the generated operation sequence to achieve a more comprehensive detection of the target program. Among them, because the second operation sequence is automatically expanded by the neural network model, it is beneficial to improve the detection efficiency.

[0028] First, the Figure 1 , 2 , and 3 will be used to introduce the application scenarios involved in the embodiments of the present application.

[0029] In the embodiments of the present application, the provided information processing method can be executed by an electronic device. In this way of being executed by the electronic device, all steps in the information processing method provided in the embodiments of the present application can be executed by the electronic device. For example, as Figure 1 shown, all steps in the information processing method provided in the embodiments of the present application can be executed by the program for performing detection in the electronic device 100. In this case, the program for performing detection can detect the target program in the electronic device through the information processing method provided in the embodiments of the present application.

[0030] Alternatively, the information processing method provided in the embodiments of the present application can also be executed by a server. Correspondingly, in this way of being executed by the server, the server can start to execute the steps in the information processing method provided in the embodiments of the present application in response to a trigger instruction. Among them, the trigger instruction can be sent by the electronic device used by the user, or can be locally triggered by the server in response to some automated events.

[0031] In addition, the information processing method provided in the embodiments of the present application can also be executed jointly by an electronic device and a server. In this way of being executed jointly by the electronic device and the server, some steps in the information processing method provided in the embodiments of the present application are executed by the electronic device, while other steps are executed by the server. Exemplarily, as Figure 2As shown, the electronic device 100 can execute the steps included in the information processing method: perform a first operation process on the target program and record the executed first operation process. After that, the electronic device 100 can transmit the first operation process to the server 200, and the server 200 can obtain the corresponding first operation sequence through the first operation process, and then execute a second operation process on the target program through the first operation sequence, interface information, and neural network model. Among them, the server 200 executing the second operation process on the target program can be understood as the server 200 sending an operation instruction to the electronic device 100, so that the electronic device 100 executes the second operation sequence on the target program based on the simulated operation method and detects the execution process.

[0032] It should be noted that in this way of collaborative execution by the electronic device and the server, the steps respectively executed by the electronic device and the server are not limited to the methods described in the above examples. In actual applications, the steps respectively executed by the electronic device and the server can be dynamically adjusted according to the actual situation. In addition, in this way of collaborative execution by the electronic device and the server, it can also be the collaborative execution of multiple electronic devices and the server. For example, as Figure 3 shown, the first operation process can be collected by the electronic device 100 and transmitted to the server 200. Then, after the server 200 obtains the second operation sequence through the first operation sequence, interface information, and neural network model, it can execute the second operation sequence on the target program in the electronic device 110. Optionally, the electronic device 100 can be the device actually used by the user, and the electronic device 110 can be the device used for program detection.

[0033] It should be noted that in addition to being Figures 1-3 the smart phone shown in, the electronic device 100 can also be a device such as a tablet computer. The server 200 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers.

[0034] The following will specifically describe the embodiments of the present application with reference to the accompanying drawings.

[0035] Please refer to Figure 4 , an information processing method provided by an embodiment of the present application, the method includes:

[0036] S110: Obtain the corresponding first operation sequence by performing the first operation process on the target program.

[0037] In the embodiment of the present application, the target program can be understood as the program to be detected. Among them, there can be one target program, or multiple target programs.

[0038] Optionally, the first operation process can be a process in which the user manually operates the target program. During the operation of the target program, a certain function of the target program can be triggered through continuous operations, or the target program can be triggered to switch and display to a certain interface through continuous operations.

[0039] As a way, the obtained first operation sequence can be understood as a process representing the continuous operations. Optionally, the first operation sequence includes the interface operated each time and the control operated in the interface each time. Exemplarily, the obtained first operation sequence can include: clicking on control C11 in interface P1, clicking on control C21 in interface P2, clicking on control C31 in interface P3, and clicking on control C41 in interface P4, etc. Among them, by clicking on control C11 in interface P1, the target program can be triggered to display interface P2, by clicking on control C21 in interface P2, the target program can be triggered to display interface P3, and by clicking on control C31 in interface P3, the target program can be triggered to display interface P4.

[0040] Optionally, the first operation sequence includes the interface operated each time and the position operated in the interface each time. Exemplarily, the obtained first operation sequence can include: clicking on position W11 in interface P1, clicking on position W21 in interface P2, clicking on position W31 in interface P3, and clicking on position W41 in interface P4, etc. Among them, by clicking on position W11 in interface P1, the target program can be triggered to display interface P2, by clicking on position W21 in interface P2, the target program can be triggered to display interface P3, and by clicking on position W41 in interface P3, the target program can be triggered to display interface P4.

[0041] In the embodiments of the present application, there are multiple ways to obtain the first operation sequence.

[0042] As a way, screen recording can be performed when the first operation process is executed on the target program, so that the video data obtained by screen recording can include the first operation process. In this case, the screen recording data can be recognized to obtain the first operation sequence. Optionally, when performing screen recording when the first operation process is executed on the target program, a preset identifier can be displayed at the operation position each time a touch operation acting on the screen is detected, so that later when identifying the operation position in each interface through the recorded data obtained by screen recording, the preset identifier can be used to assist in determining the operation position of the user.

[0043] As a way, when the first operation process is executed on the target program, each operation executed can be recorded, and then the first operation sequence can be obtained through the recorded multiple operations.

[0044] In an embodiment of the present application, the first operation process may be implemented by a user on the electronic device they use. As a way, the electronic device may start obtaining the first operation sequence when detecting the startup of a target program. For example, in the case of obtaining the first operation sequence by means of screen recording, the electronic device may start screen recording when detecting the startup of the target program, and then end the screen recording when the target program exits to obtain screen recording data. Additionally, in an embodiment of the present application, during the running of the target program, if it is detected that no operation on the target program has been detected for a long time, the screen recording may be paused to avoid having many duplicate video frames in the screen recording data.

[0045] S120: Execute a second operation process on the target program through the first operation sequence, interface information, and neural network model, where the second operation process includes performing operations on the target program based on a second operation sequence, and the second operation sequence is obtained by the neural network model based on the interface information and the first operation sequence, and the interface information includes information about the interface corresponding to the first operation sequence.

[0046] Among them, in addition to obtaining the operation sequence, relevant information about the interfaces operated during the execution of the first operation process will also be obtained to obtain the interface information. Among them, the interface information including the information about the interface corresponding to the first operation sequence can be understood as: the interface information may include the information in the interfaces involved in the first operation sequence, where the interfaces involved in the first operation sequence can be understood as the interfaces operated in the operations included in the first operation sequence.

[0047] Exemplarily, the first operation sequence may include: clicking on the control C11 in the interface P1, clicking on the control C21 in the interface P2, clicking on the control C31 in the interface P3, and clicking on the control C41 in the interface P4, etc. In this case, among the operations included in the first operation sequence, the interfaces operated may include the interfaces P1, P2, P3, P4, and further, the interfaces corresponding to this first operation sequence include the interfaces P1, P2, P3, P4.

[0048] As a way, the interface content features of each operation in the first operation sequence may be obtained to obtain an interface content feature sequence corresponding to the first operation sequence, and the interface content feature sequence may be used as the interface information. Optionally, a target detection algorithm may be used to obtain the interface content features of each operation in the first operation sequence. For example, the effect of identifying the interface through the target detection algorithm may be as Figure 5 shown. And because the target detection algorithm is adopted, the recognition ability for web pages can be increased, which is beneficial to increasing the comprehensiveness of detection.

[0049] Among them, the interface content feature sequence represents the correlation between interface content features and timestamps. In this case, in the interface content feature sequence, the interfaces to which adjacent interface content features (adjacent in the time dimension) belong are also the interfaces that are operated on adjacent in the first operation sequence. Exemplarily, as Figure 6 shown, the interfaces involved in the first operation sequence include interface 10, interface 11, interface 12, and interface 13. Among them, in the first operation process, the time to operate on interface 10 is T1, the time to operate on interface 11 is T2, the time to operate on interface 12 is T3, and the time to operate on interface 13 is T4. After obtaining the interface content features of each operation in the first operation sequence, the interface content features can be associated with their respective timestamps. Among them, the timestamp corresponding to the interface content feature represents the operation time of the interface to which the interface content feature belongs. Based on this example, the interface content feature Z1 of interface 10 can be associated with time T1, the interface content feature Z2 of interface 11 can be associated with time T2, the interface content feature Z3 of interface 12 can be associated with time T3, and the interface content feature Z4 of interface 13 can be associated with time T4. Thus, the obtained interface content feature sequence includes interface content feature Z1, interface content feature Z2, interface content feature Z3, and interface content feature Z4.

[0050] Optionally, the interface content features include the content in the interface and the interface layout features. Among them, the content in the interface includes the text information displayed in the interface (such as prompt information, button labels, etc.), the controls included, the icons included, etc.

[0051] In the embodiments of the present application, the neural network model has image recognition capabilities and natural language processing capabilities, and can thus identify the input interface information to expand the first operation sequence. Optionally, the neural network model in the embodiments of the present application can be a large language model (LLM).

[0052] Among them, the large language model is a model based on machine learning and natural language processing technologies. It learns the ability of language understanding and generation through training on a large amount of text data. The large language model is huge in scale, usually containing billions or even trillions of parameters, which enables them to capture the nuances and complexities of language, thereby generating more natural and fluent text. In one way, the large language model can adopt a multi-layer Transformer structure, which makes it perform excellently in processing natural language tasks. The Transformer architecture is a model based on the self-attention mechanism and can process sequence data such as text. The self-attention mechanism allows the model to consider other words in the entire text sequence when processing a word, so as to better understand the context. The large language model can generate coherent and meaningful statements according to the given prompt or context. This ability is not limited to simple sentence generation, but can also generate complex articles, conversations, etc. (for example, it can generate the second operation sequence in the embodiments of the present application).

[0053] In the embodiments of the present application, after obtaining the first operation sequence and the interface information, the first operation sequence and the interface information can be input into the neural network model. Among them, after receiving the first operation sequence and the interface information, the neural network model can expand the first operation sequence through the interface information to obtain the second operation sequence. It should be noted that the operation sequence can also be understood as an operation path. One or more operation paths are involved in the first operation sequence. However, the operation paths included in the first operation sequence may not be all the operation paths that the target program can implement. Therefore, in the embodiments of the present application, the neural network model can be used to expand more possible operation paths to obtain the second operation sequence.

[0054] As a way, the first operation sequence refers to a set of operation instructions executed by the user on the target program. These instructions are usually arranged in a specific order and are intended to achieve a certain function or reach a certain target state. The interface information details the structure, element layout, and functional attributes of each element of the application interface or software interface with which the user interacts. These information are crucial for the neural network model to understand the context of the user's operations. In this scenario, after the input layer of the neural network model receives the first operation sequence and the interface information, it will process these information through the encoding layer and convert them into a representation form that can be understood inside the neural network model. The process of expanding the first operation sequence can be regarded as the neural network model predicting or generating possible subsequent operations, or other operations on the basis of understanding the current operation and the context of its interface. In the embodiments of the present application, the expansion of the first operation sequence is not just simple operation copying or random addition, but an intelligent inference based on the logical relationship of interface elements, the user's historical behavior pattern, and possible program logic constraints.

[0055] After obtaining the first operation sequence, the second operation process can be performed on the target program based on the second operation sequence. Among them, performing the second operation process on the target program based on the second operation sequence can be understood as sequentially implementing the operations included in the second operation sequence on the target program. Exemplarily, the second operation sequence may include: clicking on the control C11 in the interface P1, clicking on the control C22 in the interface P2, clicking on the control C51 in the interface P5, and clicking on the control C61 in the interface P6, etc. In this case, the control C11 in the interface P1 of the target program can be clicked in the way of simulating operations to trigger the display of the interface P2 of the target program, then the control C22 in the interface P2 can be clicked in the way of simulating operations, then the control C51 in the interface P5 can be clicked in the way of simulating operations, and then the control C61 in the interface P6 can be clicked in the way of simulating operations, so as to complete the execution of the second operation sequence on the target program.

[0056] Among them, as a way, performing the second operation process on the target program through the first operation sequence, interface information, and neural network model can be understood as inputting the first operation sequence and interface information into the neural network model, so that the neural network model generates the second operation sequence and performs the second operation sequence on the target program. Among them, the neural network model can perform the second operation process on the target program after completing the generation of the second operation sequence, or can perform operations on the target program while generating the second operation sequence through the already generated operation sequence (a partial operation sequence in the second operation sequence).

[0057] In the embodiment of the present application, after obtaining the first operation sequence, the operation entry position can also be identified through the first operation sequence. Among them, the operation entry position can be understood as the position where the operation starts in the first operation sequence. The operation entry position can be a control in an interface of the target program, or can be a position in the interface. In this way, the first operation sequence, the entry position, and the interface information can be input into the neural network model, so that the neural network model generates the second operation sequence.

[0058] As a way, the neural network model can trigger the execution of the second operation sequence on the target program after generating the second operation sequence. Or, as a way, the neural network model can, during the process of generating the second operation sequence, perform the already generated operation sequence on the target program in real time, without waiting for the second operation sequence to be completely generated before performing the second operation sequence on the target program. In addition, the neural network model can trigger the execution of the second operation sequence on the target program. In this case, it can be understood that the neural network model is simulating the operations of the user and performing the second operation sequence on the target program.

[0059] S130: Detect the second operation process.

[0060] Among them, the executed second operation process can also be understood as a kind of inspection process. The so-called inspection can be abbreviated as a process of simulating manual operations to operate the target program. In the embodiment of the present application, the purpose of executing the second operation sequence on the target program is to detect whether there are potential errors in the target program. Therefore, during the process of triggering the execution of the second operation sequence on the target program, the second operation process will also be detected synchronously.

[0061] Among them, the occurrence of an error in the target program can include that the interface cannot be displayed normally, or the displayed content is incorrect, etc. Therefore, during the process of supporting the second operation series for the target program, if an error occurs in the target program, then there will be a corresponding manifestation in the content displayed by the target program. Therefore, as a way, it is possible to synchronously collect images of the target program during the execution of the second operation sequence, so as to detect the second operation process through the synchronously collected images.

[0062] Optionally, when starting to execute the second operation process, start screen recording synchronously, so as to collect the content displayed by the target program during the second operation process in the form of screen recording, in order to detect the second operation process. Optionally, during the execution of the second operation process, take one or more consecutive screenshots of the screen after each operation, so as to detect the second operation process through the screenshots.

[0063] As a way, the method further includes: if an abnormality is detected during the execution of the second operation process, output the optimization suggestion content corresponding to the abnormality. Among them, the possible abnormalities of the target program include: such as error prompts, repeated operations, and unexpected jumps. When an abnormality is detected during the execution of the second operation process, corresponding optimization suggestions can be given for the detected abnormality. Optionally, an optimization plan can be automatically generated, such as adjusting the interface layout, optimizing the interaction path, or prompting the developer to fix code defects.

[0064] Optionally, the inspection results (the detection results of the second operation process) and the domain knowledge base can be combined to generate adaptive optimization suggestions, providing clear guidance for interface design and interaction process improvement, and forming a closed-loop process from detection to optimization. In the embodiment of the present application, the corresponding relationship between abnormalities and optimization suggestions is stored in the domain knowledge base.

[0065] During the detection of the second operation process, if a problem is found, a complete report including a description of the key problem, a screenshot of the problem frame, root cause analysis, and optimization suggestions can be generated to facilitate the developer's quick understanding of the problem. Among them, various data visualization forms (such as a timeline, a problem heat map) can be provided to help users intuitively understand the quality of the interface interaction and the problem distribution.

[0066] An information processing method provided in this embodiment enables, through the above-mentioned manner, after obtaining the first operation sequence corresponding to the target program, the neural network model can expand the operation sequence based on the interface information corresponding to the first operation sequence and the first operation sequence to generate more operation sequences (the second operation sequence), so that the target program can be operated based on the generated operation sequences to achieve a more comprehensive detection of the target program. Among them, since the second operation sequence is automatically expanded by the neural network model, it is beneficial to improve the detection efficiency.

[0067] Please refer to Figure 7 , an information processing method provided in an embodiment of the present application, the method includes:

[0068] S210: Obtain a video frame sequence by recording data, where the recording data is obtained by video-recording the first operation process of the target program.

[0069] In this embodiment, screen recording can be started after the target program is started to implement video recording of the first operation process of the target program. It should be noted that the video data may include multiple video frames, and the recording data in this embodiment can be understood as the video data obtained by recording. After obtaining the recording data, the recording data can be frame-divided to obtain a video frame sequence.

[0070] S220: Obtain the first operation sequence from the video frame sequence.

[0071] It should be noted that since the recording data represents the first operation process of the target program, by recognizing the video frame sequence, the operations specifically executed in the first operation process can be obtained, thereby obtaining the first operation sequence.

[0072] It should be noted that during the video recording of the first operation process of the target program, some repetitive or ineffective content may be recorded. For example, during the video recording, if the user does not trigger subsequent operations all the time, many video frames with the same content may be recorded. Another example is that during the video recording, the intermediate pictures during the interface jump or switch may also be recorded. During the interface jump or switch, a completely black picture may appear, resulting in all black video frames in the recorded video frames, and such all-black video frames have no practical effect. Therefore, as a method, obtaining the first operation sequence from the video frame sequence may include: preprocessing the video frame sequence to obtain the preprocessed video frame sequence, and then obtaining the first operation sequence from the preprocessed video frame sequence. As a method, the preprocessing includes one or more of the following multiple processing methods, and the multiple processing methods include: removing redundant video frames, duplicate video frames, improving the video frame quality, and removing invalid video frames.

[0073] Among them, removing redundant video frames includes: removing the redundant video frames in the video frame sequence.

[0074] Among them, removing duplicate video frames includes: when it is detected that there are multiple consecutive video frames with the same content in the video frame sequence, only retaining one video frame among the multiple consecutive video frames. Exemplarily, in the video frame sequence, there are video frame F1, video frame F2, video frame F3, video frame F4, and video frame F5. Among them, if the content of video frame F2 and video frame F3 is repeated, one of video frame F2 and video frame F3 can be retained, and the other video frame can be removed. For example, when retaining video frame F2, the preprocessed video frame sequence is: video frame F1, video frame F2, video frame F4, and video frame F5. Or, when retaining video frame F3, the preprocessed video frame sequence is: video frame F1, video frame F3, video frame F4, and video frame F5.

[0075] In the embodiments of the present application, there are various ways to determine whether the content of two video frames is the same.

[0076] As a method, it can be determined whether the content of two video frames is the same based on feature comparison. In this method, the feature codes (which can also be understood as feature vectors) of the two video frames can be obtained first. If the similarity of the feature codes of the two video frames is greater than the similarity threshold, it is determined that the content of the two video frames is the same. Otherwise, it is determined that the content of the two video frames is different. Optionally, the vector distance can be used to measure whether the feature codes of the two video frames are similar.

[0077] As another approach, it is possible to determine whether the content of two video frames is the same based on the pixels of the two video frames. Optionally, based on the pixels of the two video frames, a difference value between the two video frames can be calculated, and then whether the content of the two video frames is the same can be determined according to this difference value. It should be noted that for a video frame, each pixel included will have a corresponding pixel value. In this case, when calculating the difference value between two video frames, the difference value of the pixel values of two pixels with the same position in the two video frames can be calculated first to obtain the difference values of the pixel values corresponding to all pairs of pixels with the same position in the two video frames, and then the absolute values of all these pixel value differences are summed to obtain the difference value between the two video frames. If this difference value is greater than a preset difference threshold, it is determined that the content of the two video frames is different; otherwise, it is determined that the content of the two video frames is the same.

[0078] Exemplarily, as Figure 8 shown, the two video frames for which the difference value is calculated include video frame 20 and video frame 21. Among them, video frame 20 includes pixels P11, P12......P33. Correspondingly, video frame 21 includes pixels P11, P12......P33. Among them, the position of pixel P11 in video frame 20 is the first one in the first row, and the position of pixel P11 in video frame 21 is also the first one in the first row. Therefore, pixel P11 in video frame 20 and pixel P11 in video frame 21 can be understood as two pixels with the same position. By analogy, all pairs of pixels with the same position in video frame 20 and video frame 21 can be obtained. Among them, the absolute values of the difference values of the pixel values between pixel P11 in video frame 20 and pixel P11 in video frame 21, the difference values of the pixel values between pixel P12 in video frame 20 and pixel P12 in video frame 21...... the difference values of the pixel values between pixel P33 in video frame 20 and pixel P33 in video frame 21 are summed respectively, and then the difference value between video frame 20 and video frame 21 can be obtained.

[0079] Among them, the difference value of the pixel values of two pixels can be obtained by summing the absolute values of the differences of the two pixels on multiple color channels. For example, in the RGB color mode, the difference values of the two pixels on the red channel, the green channel, and the blue channel can be calculated, and then the absolute values of the differences of these channels are summed to obtain the difference value of the pixel values of the two pixels.

[0080] Optionally, if the image parameter includes pixel values, the difference between two video frames on a specified color channel can be used as the corresponding difference value. For example, for two video frames for which the difference value is to be calculated, the channel means of the two video frames on the specified color channel can be calculated, and then the difference between the respective channel means can be obtained to get the difference value. In the embodiments of the present application, during the process of calculating the channel mean of a video frame, the value of each pixel in the video frame on the specified color channel can be obtained first, and then the values of each pixel in the picture on the specified color channel are summed to obtain the total color value. By comparing the total color value with the number of pixels in the picture, the channel mean of the specified color channel can be obtained.

[0081] Optionally, for two video frames for which it is to be determined whether the comparison content is the same, they can be converted into corresponding grayscale images first and then compared. It should be noted that in a grayscale image, the brightness value of a pixel represents the gray level or brightness intensity of that pixel point. A grayscale image is an image that contains only brightness information and no color information. In a grayscale image, each pixel is usually represented by a single value (for example, a brightness value between 0 and 255), where 0 represents black, 255 represents white, and intermediate values represent different shades of gray. In this case, when calculating the difference between two pixels at the same position, only the difference in brightness values needs to be calculated, without calculating the difference in each color channel, which can help reduce the amount of calculation and improve the final detection speed. It should be noted that the method of calculating the difference value based on brightness values is the same as the method of calculating the difference value based on pixel values described above. Therefore, the method of calculating the difference value based on brightness values can refer to the foregoing content, and the method of calculating the difference value based on brightness values will not be introduced here.

[0082] Among them, improving the image quality of a video frame can include improving the clarity of each video frame in the video frame sequence, so as to be more accurate in subsequent recognition.

[0083] Among them, removing invalid video frames can include removing video frames with solid-color content. For example, it can include removing video frames with pure black content and removing video frames with pure white content.

[0084] S230: Execute a second operation process on the target program through the first operation sequence, interface information, and neural network model. Among them, the second operation process includes performing operations on the target program based on the second operation sequence, and the second operation sequence is obtained by the neural network model based on the interface information and the first operation sequence. The interface information includes information about the interface corresponding to the first operation sequence.

[0085] S240: Detect the second operation process.

[0086] It should be noted that the duration of the first operation process of the target program may be short or long. Among them, in the case of a long duration, the recorded data obtained by screen recording will also be relatively long. In this case, the recorded data can be segmented to obtain multiple video frame sequences, so that each video frame sequence can be recognized separately. Correspondingly, as a method, obtaining video frame sequences from the recorded data may include: segmenting the recorded data by a preset method to obtain multiple video frame sequences.

[0087] Among them, in the embodiments of the present application, there are various ways to segment the recorded data. As a method, the recorded data can be segmented by a preset duration to obtain multiple video frame sequences. In this method, the segmentation points can be selected from the start position of the recorded data, where the interval duration between two adjacent segmentation points is the preset duration.

[0088] In the embodiments of the present application, there are various ways to determine the preset duration.

[0089] Optionally, the preset duration can be pre-configured by the developer. Optionally, the preset duration can be determined by the operation complexity of the target program. Among them, the operation complexity of the target program can be determined according to the functions included in the target program and the interfaces included in the target program. Among them, the more functions included in the target program, the higher the operation complexity of the target program. Correspondingly, the more interfaces included in the target program, the higher the operation complexity of the target program.

[0090] Optionally, the recorded data can be segmented by scene change points to obtain multiple video frame sequences. Among them, the scene change point can be understood as the point where the operation scene changes, and this point can be understood as a video frame. Therefore, segmenting the recorded data by scene change points can also be understood as segmenting the recorded data based on the video frame where the scene changes.

[0091] Exemplarily, as Figure 9 shown, the video frames included in the recorded data include video frame 20, video frame 21, video frame 22, video frame 23, video frame 24, video frame 25, video frame 26, and video frame 27. Among them, if it is detected that video frame 23 is the scene change point, the recorded data can be segmented based on video frame 23. For example, video frames 20, 21, and 22 can be segmented into one video frame sequence, and video frames 23, 24, 25, 26, and 27 can be segmented into one video frame sequence, so as to obtain Figure 9 the video frame sequences X1 and X2 shown in

[0092] Correspondingly, in the case where there are multiple video frame sequences, obtaining the first operation sequence from the video frame sequences may include: obtaining the first operation sequence from the multiple video frame sequences, where the first operation sequence includes operation sequences corresponding to the multiple video frame sequences respectively. Exemplarily, taking the multiple video frame sequences obtained in Figure 9 as an example, based on the situation shown in Figure 9 , the corresponding operation sequence can be obtained for the video frame sequence X1, and the corresponding operation sequence can be obtained for the video frame sequence X2. Furthermore, in the obtained first operation sequence, the operation sequence corresponding to the video frame sequence X1 and the operation sequence corresponding to the video frame sequence X2 can be included.

[0093] An information processing method provided in this embodiment enables, through the above method, after obtaining the first operation sequence corresponding to the target program, the neural network model to expand the operation sequence based on the interface information corresponding to the first operation sequence and the first operation sequence to generate more operation sequences (second operation sequences), so that the target program can be operated based on the generated operation sequences to achieve a more comprehensive detection of the target program. Among them, since the second operation sequence is automatically expanded by the neural network model, it is beneficial to improve the detection efficiency. And in this embodiment, the first operation process of the target program can be video-recorded, and then the subsequent first operation sequence can be generated from the recorded video data, which is beneficial to make the obtained first operation sequence more complete.

[0094] Moreover, in this embodiment, an intelligent patrol inspection model based on screen recording frame division is provided, which uses a large language model to efficiently analyze and intelligently process the screen recording content, and is applicable to scenarios such as automated system patrol inspection, interface anomaly detection, and interaction process optimization. Among them, in this embodiment, the screen recording data can be sliced into continuous video frame sequences along the time axis through an automatic frame division technology, and combined with an optical character recognition technology and an image analysis technology, the interface elements (such as texts, icons, controls) of each frame are accurately extracted, so as to obtain the starting entrance of the patrol inspection and the basic operation steps for obtaining the entrance, providing accurate input for subsequent analysis.

[0095] Furthermore, in this embodiment, the large language model is innovatively applied to the context analysis of the frame sequence content. And an object detection algorithm is innovatively integrated into the traditional element recognition technology to make up for the blind area where local recognition is currently impossible, improving the coverage of intelligent patrol inspection. Moreover, in this embodiment, the adaptive optimization suggestions can be generated by combining the patrol inspection results and the domain knowledge base, providing clear guidance for interface design and interaction process improvement, and forming a closed-loop process from detection to optimization.

[0096] Please refer to Figure 10 , an information processing method provided in an embodiment of this application, the method includes:

[0097] S310: Obtain a video frame sequence from the recorded data, where the recorded data is obtained by video recording the first operation process of the target program.

[0098] S320: Obtain a first operation sequence from the video frame sequence.

[0099] S330: Obtain the interface content features of the interface included in each video frame in the video frame sequence to obtain an interface content feature sequence corresponding to the first operation sequence.

[0100] S340: Execute a second operation process on the target program through the first operation sequence, the interface content feature sequence, and the neural network model, where the second operation process includes performing operations on the target program based on a second operation sequence, and the second operation sequence is obtained by the neural network model based on the interface content feature sequence and the first operation sequence.

[0101] S350: Detect the second operation process.

[0102] An information processing method provided in this embodiment can improve the efficiency of detection in the above manner. Moreover, in this embodiment, after obtaining the video frame sequence, the interface content features of the interface included in each video frame in the video frame sequence can be obtained to obtain an interface content feature sequence corresponding to the first operation sequence, so that the second operation sequence extended by the neural network model can be more rich and accurate, which is conducive to improving the comprehensiveness of the inspection of the target operation program.

[0103] Next, Figure 11 the steps of an information processing method according to an embodiment of the present application will be described. As Figure 11 shown, the overall process can be divided into three parts: identifying element information, generating an operation stack, and detection. Among them, identifying element information can be understood as identifying the information of elements in each video frame in the frame sequence to obtain interface information. The process of generating an operation stack can be understood as the process of generating the first operation sequence described above. The detection process can be understood as the process of executing the second operation process on the target program through the first operation sequence, the interface information, and the neural network model, and detecting the second operation process.

[0104] In an embodiment of the present application, it may be to analyze the first operation process in multiple electronic devices to obtain a second operation sequence. In this case, the steps of the information processing method provided by the embodiment of the present application may include: obtaining a first operation sequence corresponding to the first operation process in multiple electronic devices through the first operation process performed on the target program in multiple electronic devices, so as to obtain multiple first operation sequences. Performing a second operation process on the target program through multiple first operation sequences, interface information, and a neural network model, where the second operation process includes performing an operation on the target program based on the second operation sequence, and the second operation sequence is obtained by the neural network model based on the interface information and the first operation sequence, and the interface information includes information of the interface corresponding to the first operation sequence. Detecting the second operation process.

[0105] Please refer to Figure 12 , an information processing apparatus 400 provided by an embodiment of the present application, the apparatus 400 includes:

[0106] An operation sequence acquisition unit 410, configured to obtain a corresponding first operation sequence by performing a first operation process on the target program.

[0107] An operation execution unit 420, configured to perform a second operation process on the target program through the first operation sequence, interface information, and a neural network model, where the second operation process includes performing an operation on the target program based on the second operation sequence, and the second operation sequence is obtained by the neural network model based on the interface information and the first operation sequence, and the interface information includes information of the interface corresponding to the first operation sequence.

[0108] A detection unit 430, configured to detect the second operation process.

[0109] As a way, the operation sequence acquisition unit 410 is specifically configured to obtain a video frame sequence through recorded data, where the recorded data is obtained by video-recording the first operation process of the target program; and obtain the first operation sequence through the video frame sequence.

[0110] Optionally, the operation sequence acquisition unit 410 is specifically configured to segment the recorded data by a preset method to obtain multiple video frame sequences, and obtain the first operation sequence through the multiple video frame sequences, where the first operation sequence includes operation sequences corresponding to the multiple video frame sequences respectively.

[0111] Optionally, the operation sequence acquisition unit 410 is specifically configured to segment the recorded data by a preset duration to obtain multiple video frame sequences; or segment the recorded data by scene change points to obtain multiple video frame sequences.

[0112] Optionally, the operation sequence acquisition unit 410 is specifically configured to preprocess the video frame sequence to obtain a preprocessed video frame sequence; and obtain a first operation sequence through the preprocessed video frame sequence, where the preprocessing includes one or more of the following multiple processing methods, and the multiple processing methods include: removing duplicate video frames, improving the video frame image quality, and removing invalid video frames.

[0113] As a method, the operation execution unit 420 is further configured to obtain the interface content feature of the interface of each operation in the first operation sequence, so as to obtain an interface content feature sequence corresponding to the first operation sequence, and use the interface content feature sequence as interface information.

[0114] As a method, the detection unit 430 is further configured to output optimization suggestion content corresponding to an exception if an exception occurs during the execution of the second operation.

[0115] An information processing device provided in this embodiment, so that after obtaining the first operation sequence corresponding to the target program through the above information processing device, the neural network model can expand the operation sequence based on the interface information corresponding to the first operation sequence and the first operation sequence to generate more operation sequences (second operation sequences), so as to operate the target program based on the generated operation sequences to achieve a more comprehensive detection of the target program. Among them, since the second operation sequence is automatically expanded by the neural network model, it is beneficial to improve the detection efficiency.

[0116] It should be noted that the device embodiment in this application corresponds to the foregoing method embodiment. The specific principle in the device embodiment can refer to the content in the foregoing method embodiment, and will not be elaborated here.

[0117] Next, Figure 13 an electronic device provided in this application will be described.

[0118] Please refer to Figure 13 , based on the foregoing information processing method and device, another electronic device 1000 that can execute the foregoing information processing method is further provided in an embodiment of this application. The electronic device 1000 includes one or more (only one is shown in the figure) processors 102, a memory 104, a network module 106, a screen 108, a sensor module, and an audio collection device that are coupled to each other. Among them, a program that can execute the content in the foregoing embodiment is stored in the memory 104, and the processor 102 can execute the program stored in the memory 104.

[0119] Among them, the processor 102 may include one or more processing cores. The processor 102 connects various parts within the entire electronic device 1000 through various interfaces and lines. By running or executing instructions, programs, code sets, or instruction sets stored in the memory 104, and by invoking the data stored in the memory 104, it performs various functions of the electronic device 1000 and processes data. Optionally, the processor 102 may be implemented in at least one of the hardware forms of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). Alternatively, the processor 102 may be an application processor. The processor 102 may integrate a combination of one or several of a central processing unit (CPU), a graphics processing unit (GPU), and a modem, etc. Among them, the CPU mainly processes the operating system, user interface, application programs, etc.; the GPU is responsible for rendering and drawing the displayed content; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 102 and may be implemented separately through a communication chip.

[0120] The memory 104 may include a random access memory (RAM) and may also include a read-only memory. The memory 104 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 104 may include a program storage area and a data storage area. Among them, the program storage area may store instructions for implementing the operating system, instructions for implementing at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the following various method embodiments, etc.

[0121] The network module 106 is used to implement information interaction between the electronic device 1000 and other devices. For example, it transmits device control instructions, manipulation request instructions, and status information acquisition instructions, etc. When the electronic device 1000 is specifically different devices, the corresponding network module 106 may be different.

[0122] The screen 108 can be used for screen display. For example, when a game program is running, it can be used to display the game screen corresponding to the game program. In one way, the electronic device 1000 can capture multiple consecutive frames of the screen 108 to obtain multiple frames of the screen required for detecting whether the electronic device 100 is displaying an event animation.

[0123] The sensor module can include at least one sensor. Specifically, the sensor module can include but is not limited to: a light sensor, a motion sensor, a pressure sensor, an infrared thermal sensor, a distance sensor, an acceleration sensor, and other sensors.

[0124] Among them, the pressure sensor can detect the pressure generated by pressing on the electronic device 1000. That is, the pressure sensor detects the pressure generated by the contact or pressing between the user and the audio playback device, such as the pressure generated by the contact or pressing between the user's ear and the mobile terminal. Therefore, the pressure sensor can be used to determine whether there is contact or pressing between the user and the electronic device 1000, and the magnitude of the pressure.

[0125] Among them, the acceleration sensor can detect the magnitude of acceleration in each direction (generally three axes). When stationary, it can detect the magnitude and direction of gravity, and can be used in applications for identifying the posture of the electronic device 1000 (such as horizontal and vertical screen conversion, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometers, taps), etc. In addition, the electronic device 1000 can also be configured with other sensors such as gyroscopes, barometers, hygrometers, thermometers, etc., which will not be elaborated here.

[0126] The audio acquisition device is used for audio signal acquisition. Optionally, the audio acquisition device includes a plurality of audio acquisition components. The audio acquisition component can be a microphone.

[0127] As one way, the network module of the electronic device 1000 is a radio frequency module. This radio frequency module is used to receive and send electromagnetic waves, realize the mutual conversion between electromagnetic waves and electrical signals, so as to communicate with a communication network or other devices. The radio frequency module can include various existing circuit components for performing these functions. For example, antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, user identity module (SIM) cards, memories, etc. For example, the radio frequency module can interact with external devices through the sent or received electromagnetic waves, and then receive the audio signals sent by the external devices.

[0128] Alternatively, the electronic device 1000 may further include an image acquisition device for image acquisition. For example, videos, static pictures, or dynamic pictures can be captured by the image acquisition device. Additionally, the electronic device 1000 may further include a motor driver IC and a motor. The motor can drive the electronic device 1000 to perform a conversion of the device state. For example, in the case where the electronic device 1000 includes a rollable screen, the motor can drive the rollable screen to contract or expand, so as to cause the electronic device 1000 to perform a conversion of the device state.

[0129] Please refer to Figure 14 , which shows a structural block diagram of a computer-readable storage medium provided by an embodiment of the present application. Program code is stored in the computer-readable medium 1100, and the program code can be called by a processor to execute the method described in the above method embodiment.

[0130] The computer-readable storage medium 1100 may be an electronic memory such as a flash memory, EEPROM (electrically erasable programmable read-only memory), EPROM, a hard disk, or a ROM. Optionally, the computer-readable storage medium 1100 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 1100 has a storage space for the program code 1110 that executes any method step in the above method. These program codes can be read from or written into one or more computer program products. The program code 1110 can be compressed in an appropriate form, for example.

[0131] In summary, an information processing method, apparatus, and electronic device provided by the present application. In this method, after obtaining a corresponding first operation sequence through a first operation process performed on a target program, a second operation process can be performed on the target program through the first operation sequence, interface information, and a neural network model. The second operation process includes performing an operation on the target program based on a second operation sequence, and the second operation sequence is obtained by the neural network model based on the interface information and the first operation sequence, and the second operation process is detected. Thus, in the above manner, after obtaining the first operation sequence corresponding to the target program, the neural network model can expand the operation sequence based on the interface information corresponding to the first operation sequence and the first operation sequence to generate more operation sequences (the second operation sequence), so that the target program can be operated based on the generated operation sequence to achieve a more comprehensive detection of the target program. Since the second operation sequence is automatically expanded by the neural network model, it is beneficial to improve the detection efficiency.

[0132] Among them, in the embodiments of the present application, a technology for frame-by-frame analysis of screen recording videos is also protected. This technology automatically cuts the video data obtained by screen recording into consecutive frames and extracts the key information (such as text, icons, layout) in each frame as interface information.

[0133] Among them, in the embodiments of the present application, a method for multi-modal data analysis based on a large language model is protected. The large language model is used to understand the context of the text information and time series data in the interface information or video frame sequence, and continuously expand the possible operation paths to obtain a second operation sequence.

[0134] Among them, in the embodiments of the present application, a technology for combining inspection results and a domain knowledge base to generate targeted optimization suggestions is protected, ensuring that the inspection not only discovers problems but also provides specific improvement solutions, forming a closed-loop quality improvement mechanism.

[0135] Among them, in the embodiments of the present application, an intelligent inspection system is protected. This system includes functional modules such as screen recording data collection, frame-by-frame processing, large language model analysis (for example, using the large language model to expand the operation sequence), anomaly detection, and optimization suggestion generation, and supports automated deployment and real-time inspection to improve the inspection efficiency and accuracy. Among them, the embodiments of the present application support automated deployment, perform real-time analysis on large-scale screen recording data, significantly reduce the cost of manual intervention, and are more suitable for efficient detection of dynamic scenarios compared to traditional static interface testing methods. The embodiments of the present application are highly versatile in design and can be adapted to the interface inspection requirements of various fields, such as software quality management, user experience optimization, and operation process specification review, and have broad industrial application potential.

[0136] Among them, in the embodiments of the present application, a method for automatically generating an intelligent inspection report is protected, which can generate a detailed report including problem descriptions, solution suggestions, and data visualization (such as problem heat maps, time axes, etc.) based on the analysis results to help developers quickly locate and fix problems.

[0137] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and are not intended to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An information processing method, characterized in that: The method comprises: By executing a first operation process on the target program, a corresponding first operation sequence is obtained; Performing a second operation process on the target program through the first operation sequence, the interface information and the neural network model, wherein the second operation process includes performing an operation on the target program based on a second operation sequence, the second operation sequence is obtained by the neural network model based on the interface information and the first operation sequence, and the interface information includes information of an interface corresponding to the first operation sequence; The second operation process is detected.

2. The method according to claim 1, characterized in that The first operation sequence corresponding to the first operation process performed on the target program is obtained, including: Obtaining a video frame sequence by recording data, wherein the recording data is obtained by performing video recording on a first operation process of the target program; A first operation sequence is obtained through the video frame sequence.

3. The method according to claim 2, characterized in that Obtaining a video frame sequence from the recorded data, comprising: segmenting the recorded data in a preset manner to obtain a plurality of video frame sequences; The obtaining a first operation sequence through the video frame sequence includes: A first operation sequence is obtained through the multiple video frame sequences, and the first operation sequence includes operation sequences corresponding to each of the multiple video frame sequences.

4. The method according to claim 3, characterized in that: The step of segmenting the recorded data in a preset manner to obtain a plurality of video frame sequences includes: Segmenting the recorded data according to a preset duration to obtain a plurality of video frame sequences; Alternatively, the recorded data is segmented according to scene change points to obtain a plurality of video frame sequences.

5. The method according to claim 2, characterized in that: The obtaining a first operation sequence through the video frame sequence includes: Preprocessing the video frame sequence to obtain a preprocessed video frame sequence; Obtaining a first operation sequence through the preprocessed video frame sequence; The preprocessing includes one or more of the following processing methods, and the multiple processing methods include: removing duplicate video frames, improving video frame quality, and removing invalid video frames.

6. The method according to claim 1, characterized in that The method further comprises: The interface content features of the interface of each operation in the first operation sequence are acquired to obtain an interface content feature sequence corresponding to the first operation sequence, and the interface content feature sequence is used as interface information.

7. The method according to claim 6, characterized in that The interface content features include the content in the interface and the interface layout features.

8. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: If an abnormality is detected during the execution of the second operation, optimization suggestion content corresponding to the abnormality is output.

9. An electronic device, characterized in that: including a processor and a memory; One or more programs are stored in the memory and configured to be executed by the processor to implement the method according to any one of claims 1 to 8.

10. A computer program product, wherein the computer program product stores at least one program code, wherein the at least one program code is used to be executed by a processor to implement the method according to any one of claims 1 to 8.

Images