Method and device for generating log video of user interaction interface, equipment and medium

Abstract

The invention provides a log video generation method and device for a user interaction interface, equipment and a medium, and the method comprises the steps: carrying out the video recording of the user interaction interface; recording a running log automatically executed by a user interaction interface in a video recording process, and exporting the running log as a subtitle file; determining context information associated with the error reporting information in the subtitle file; and cutting the recorded video according to the context information to generate a log video. Through the embodiment of the invention, the abnormal information existing in the running log can be displayed more intuitively, and the processing efficiency and reliability of the running log are improved.

Description

Background Art

[0002] Currently, when using Selenium or other automation tools for UI automation, it is necessary to record the exception information during the operation process for analysis.

[0003] In the related art, a screenshot of the current interface is taken and saved in a specified directory of the project. Finally, after the project code is executed in Jenkins, the screenshot and the test report are sent to a specified mailbox.

[0004] However, in some automation scenarios, the links are long and the errors are highly correlated. Single screenshots can often only be compared with normal screenshots, and the log records where errors occur need to be captured separately.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0006] The purpose of the present disclosure is to provide a method, device, equipment and medium for generating a log video of a user interaction interface, which is used to at least overcome the problem of unintuitive error information of UI automation caused by limitations and defects of related technologies to a certain extent.

[0007] According to a first aspect of an embodiment of the present disclosure, a method for generating a log video of a user interaction interface is provided, comprising: recording a video of the user interaction interface; recording a running log of the automatic execution of the user interaction interface during the video recording process, and exporting it as a subtitle file; determining context information associated with the error information in the subtitle file; and cutting the recorded video according to the context information to generate the log video.

[0008] In an exemplary embodiment of the present disclosure, before recording the video of the user interaction interface, the method further includes:

[0009] Calling the grab function to grab the user interaction interface to be recorded, so as to determine the size information of the user interaction interface;

[0010] Call the VideoWriter_fourcc function to set the suffix of the video file to be recorded, and the suffix is ​​used to set the video file to MP4 format;

[0011] Call the VideoWriter function to set the configuration parameters of the video recording, where the configuration parameters include at least one of the encoder, frame rate, and video length and width.

[0012] In an exemplary embodiment of the present disclosure, recording a video of a user interaction interface includes:

[0013] Recording a timestamp corresponding to an initial moment of recording the user interaction interface as a first timestamp;

[0014] Call the grab function to cyclically grab the RGB image of the user interaction interface;

[0015] The RGB image is converted into an image in BGR mode of an opencv application program through a cvtColor function, and the opencv application program can perform image analysis processing on the image in BGR mode;

[0016] The video recording of the user interaction interface is stored in MP4 format, and the attribute information of the MP4 format includes the first timestamp.

[0017] In an exemplary embodiment of the present disclosure, recording the operation log of the automatic execution of the user interaction interface during the video recording process and exporting it as a subtitle file includes:

[0018] Determine a timestamp of an initial moment of the automation execution, recorded as a second timestamp;

[0019] Recording the start time and end time of recording the user interaction interface;

[0020] Recording the operation log based on the start time and the end time;

[0021] Determine a preset level to which the level of the operation log belongs, wherein the preset level includes a DEBUG level, an INFO level, a WARN level, an ERROR level, and a FATAL level;

[0022] After determining the level of the operation log, the operation log is exported as the subtitle file.

[0023] In an exemplary embodiment of the present disclosure, determining the context information associated with the error information in the subtitle file includes:

[0024] Establishing an association relationship between the subtitle file and the video in MP4 format according to the first timestamp and the second timestamp;

[0025] Determine the WARN level running log and the ERROR level running log in the subtitle file;

[0026] The context information of the error message is determined according to the WARN-level operation log and the ERROR-level operation log.

[0027] In an exemplary embodiment of the present disclosure, cutting the recorded video according to the context information to generate the log video includes:

[0028] The video in MP4 format is cut according to the error information, the context information and the configuration parameters.

[0029] In an exemplary embodiment of the present disclosure, cutting the video in MP4 format according to the error information, the context information and the configuration parameter includes:

[0030] Parsing the video in MP4 format;

[0031] Determine whether the video in MP4 format has a mask according to the parsing result;

[0032] If the video in the MP4 format has a mask, the MP4 format is saved in a video file with a mask;

[0033] Performing a cutting process on the video in the MP4 format according to the configuration parameters;

[0034] If it is determined that the videos after the cutting process do not have masks, then merging the video files after the cutting process;

[0035] If it is determined that the video after the cutting process has a mask, the concatenation result of the mask is output.

[0036] According to a second aspect of an embodiment of the present disclosure, a log video generating device of a user interaction interface is provided, comprising:

[0037] A recording module, configured to record a video of the user interaction interface;

[0038] A recording module, configured to record a running log of the automatic execution of the user interaction interface during the video recording process, and export it as a subtitle file;

[0039] A determination module, configured to determine context information associated with the error message in the subtitle file;

[0040] The cutting module is configured to cut the recorded video according to the context information to generate the log video.

[0041] According to a third aspect of the present disclosure, there is provided an electronic device, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute any one of the methods described above based on instructions stored in the memory.

[0042] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a program is stored, and when the program is executed by a processor, the log video generation method of the user interaction interface as described in any one of the above items is implemented.

[0043] In an embodiment of the present disclosure, a user interaction interface is recorded through video, and an operation log of automatic execution of the user interaction interface during the video recording process is recorded, and the operation log is exported as a subtitle file. After determining context information associated with the error information in the subtitle file, the recorded video is cut according to the context information. The generated log video is essentially a collection of log videos of error information, which can more intuitively display the abnormal information existing in the operation log, improve the processing efficiency and reliability of the operation log, and reduce the workload and data redundancy of maintaining the operation log.

[0044] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0046] Figure 1 A schematic diagram showing an exemplary system architecture of a log video generation solution of a user interaction interface to which an embodiment of the present invention can be applied;

[0047] Figure 2 is a flow chart of a log video generation method of a user interaction interface in an exemplary embodiment of the present disclosure (I);

[0048] Figure 3 is a flow chart (II) of a log video generation method of a user interaction interface in an exemplary embodiment of the present disclosure;

[0049] Figure 4 is a flow chart of a log video generation method of a user interaction interface in an exemplary embodiment of the present disclosure (III);

[0050] Figure 5 is a flow chart (IV) of a log video generation method of a user interaction interface in an exemplary embodiment of the present disclosure;

[0051] Figure 6 is a flow chart of a log video generation method of a user interaction interface in an exemplary embodiment of the present disclosure (V);

[0052] Figure 7 is a flow chart of a log video generation method of a user interaction interface in an exemplary embodiment of the present disclosure (six);

[0053] Figure 8 is a flow chart of a log video generation method of a user interaction interface in an exemplary embodiment of the present disclosure (VII);

[0054] Fig. 9 is an overall flow chart of a log video generation scheme of a user interaction interface in an exemplary embodiment of the present disclosure;

[0055] Fig.10 is a flow chart of a video recording process in a log video generation scheme of a user interaction interface in an exemplary embodiment of the present disclosure;

[0056] Fig.11 is a flowchart of obtaining a subtitle file in a log video generation solution of a user interaction interface in an exemplary embodiment of the present disclosure;

[0057] Fig.12 is a schematic diagram of a video cutting process of a log video generation solution of a user interaction interface in an exemplary embodiment of the present disclosure;

[0058] Fig.13 is a block diagram of a log video generating device of a user interaction interface in an exemplary embodiment of the present disclosure;

[0059] Fig.14 is a block diagram of an electronic device in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0060] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0061] In addition, the accompanying drawings are only schematic diagrams of the present disclosure, and the same reference numerals in the drawings represent the same or similar parts, so their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities, which do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0062] Figure 1 A schematic diagram showing an exemplary system architecture of a log video generation solution of a user interaction interface to which an embodiment of the present invention can be applied.

[0063] like Figure 1 As shown, the system architecture 100 may include one or more of terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0064] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. For example, the server 105 may be a server cluster composed of multiple servers.

[0065] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Terminal devices 101, 102, 103 can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, portable computers, desktop computers, etc.

[0066] In some embodiments, the log video generation method of the user interaction interface provided in the embodiments of the present invention is generally executed by the server 105. Accordingly, the log video generation device of the user interaction interface is generally set in the terminal device 103 (or the terminal device 101 or 102). In other embodiments, some terminals may have functions similar to those of the server device to execute the method.

[0067] The exemplary embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0068] Figure 2 1 is a flowchart of a method for generating a log video of a user interaction interface in an exemplary embodiment of the present disclosure (I).

[0069] refer to Figure 2, the log video generation method of the user interaction interface may include:

[0070] Step S202: Record the user interaction interface through video.

[0071] In an exemplary embodiment of the present disclosure, the display content of the UI interface during the automation execution process is recorded by video recording of the user interaction interface, which also includes the display content corresponding to the error message, which is more intuitive and accurate.

[0072] Step S204, recording the operation log of the automatic execution of the user interaction interface during the video recording process, and exporting it as a subtitle file.

[0073] In an exemplary embodiment of the present disclosure, by recording the operation log of the automatic execution of the user interaction interface during the video recording process, the operation log is exported as a subtitle file, that is, the machine language in the operation log is converted into a subtitle file. The subtitle file has high readability, especially the description of error information is more intuitive.

[0074] Step S206, determining context information associated with the error message in the subtitle file.

[0075] In an exemplary embodiment of the present disclosure, the location of the error message can be determined by parsing the subtitle file, and context information is determined based on the location, which helps to more comprehensively describe information related to the error message.

[0076] Step S208: cutting the recorded video according to the context information to generate the log video.

[0077] Based on the above-mentioned embodiments of the present disclosure, by performing video recording of the user interaction interface and recording the operation log of the automatic execution of the user interaction interface during the video recording process, the operation log is exported as a subtitle file, and after determining the context information associated with the error information in the subtitle file, the recorded video is cut according to the context information. The generated log video is essentially a collection of log videos of error information, which can more intuitively display the abnormal information existing in the operation log, which is conducive to operation and maintenance personnel to quickly locate the location and cause of the error information, and further improves the processing efficiency and reliability of the operation log, and reduces the workload and data redundancy of maintaining the operation log.

[0078] The following describes in detail the steps of the method for generating a log video of a user interaction interface.

[0079] Figure 3 Flowchart (II) of the method for generating a log video of a user interaction interface in an exemplary embodiment of the present disclosure.

[0080] In an exemplary embodiment of the present disclosure, Figure 3 As shown, before recording the video of the user interaction interface, the following steps are also included:

[0081] Step S302: calling the grab function to grab the user interaction interface to be recorded, so as to determine the size information of the user interaction interface.

[0082] Step S304, calling the VideoWriter_fourcc function to set the suffix of the video file to be recorded, wherein the suffix is ​​used to set the video file to the MP4 format.

[0083] In the above embodiment, the format information of the recorded video file is determined to be the MP4 format through the suffix, thereby improving the convenience of subsequent video cutting.

[0084] Step S306, calling the VideoWriter function to set the configuration parameters of the video recording, wherein the configuration parameters include at least one of an encoder, a frame rate, and a video length and width.

[0085] In the above embodiment, the reliability and compatibility of the recorded video files are improved by setting the configuration parameters of the video recording performed by UI automation.

[0086] Figure 4 Flowchart (III) of the method for generating a log video of a user interaction interface in an exemplary embodiment of the present disclosure.

[0087] In an exemplary embodiment of the present disclosure, Figure 4 As shown, video recording of the user interaction interface includes:

[0088] Step S402: Record a timestamp corresponding to the initial moment of recording the user interaction interface as a first timestamp.

[0089] Step S404, calling the grab function to cyclically grab the RGB image of the user interaction interface.

[0090] In the above embodiment, the RGB images of the user interaction interface are captured in a loop through the grab function, and all the RGB images are sequentially spliced ​​to obtain a coherent video file of the user interaction interface.

[0091] Step S406: convert the RGB image into an image in BGR mode of the opencv application program through the cvtColor function. The opencv application program can perform image analysis processing on the image in BGR mode.

[0092] In the above embodiment, by converting the RGB image into an image in the BGR mode of the opencv application, the efficiency and reliability of image analysis based on the image in the BGR mode are improved.

[0093] Step S408: storing the video recording the user interaction interface in MP4 format, wherein the attribute information of the MP4 format includes the first timestamp.

[0094] In the above embodiment, by storing the video recording the user interaction interface in MP4 format and determining the first timestamp as attribute information for matching the video file with the operation log, the reliability and timeliness of generating the log video corresponding to the error message are improved.

[0095] Figure 5 4 is a flowchart of a method for generating a log video of a user interaction interface in an exemplary embodiment of the present disclosure.

[0096] In an exemplary embodiment of the present disclosure, Figure 5 As shown, recording the operation log of the automatic execution of the user interaction interface during the video recording process and exporting it as a subtitle file includes:

[0097] Step S502: determine the timestamp of the initial moment of the automation execution, and record it as the second timestamp.

[0098] Step S504, recording the start time and end time of recording the user interaction interface.

[0099] In the above embodiment, if the automation execution and the recording are started at the same time, the second timestamp is the same as the start time.

[0100] Step S506: Record the operation log based on the start time and the end time.

[0101] Step S508, determining a preset level to which the level of the operation log belongs, wherein the preset level includes DEBUG level, INFO level, WARN level, ERROR level and FATAL level.

[0102] In the above embodiment, the meanings of the preset levels are as follows:

[0103] 1.DEBUG: The most detailed log information, usually used for debugging programs. This level is generally not used in production environments.

[0104] 2.INFO: Used to record important information during the normal operation of the program, such as system startup, shutdown, user login, etc.

[0105] 3.WARN: Warning information, indicating that the program encountered some unexpected situations during operation, but it will not cause the program to fail.

[0106] 4.ERROR: used to record abnormal situations during program operation, such as file reading and writing failure, database connection failure, etc.

[0107] 5.FATAL: The highest level error, indicating a serious error that may cause the program to be unable to continue running.

[0108] Step S510: after determining the level of the operation log, export the operation log as the subtitle file.

[0109] In the above embodiment, after determining the level of the operation log, the operation log is exported as the subtitle file, and the subtitle file also has the above level. Based on this, the subtitle file containing error information can be more efficiently determined to better record and troubleshoot problems and improve the reliability and stability of the system.

[0110] Figure 6 Flowchart (V) of the method for generating a log video of a user interaction interface in an exemplary embodiment of the present disclosure.

[0111] In an exemplary embodiment of the present disclosure, Figure 6 As shown, determining the context information associated with the error information in the subtitle file includes:

[0112] Step S602: establishing an association relationship between the subtitle file and the video in MP4 format according to the first timestamp and the second timestamp.

[0113] Step S604, determining the WARN level running log and the ERROR level running log in the subtitle file.

[0114] Step S606: determining context information of the error message according to the WARN-level operation log and the ERROR-level operation log.

[0115] In the above embodiment, the correspondence between the subtitle file and the video is determined by the first timestamp and the second timestamp, and then the video files corresponding to the WARN level operation log and the ERROR level operation log are screened, and then the video files are cut based on the error information and context information to accurately locate the error causes that lead to the WARN level operation log and the ERROR level operation log.

[0116] Figure 7 6 is a flowchart of a method for generating a log video of a user interaction interface in an exemplary embodiment of the present disclosure.

[0117] In an exemplary embodiment of the present disclosure, Figure 7 As shown, cutting the recorded video according to the context information to generate the log video includes:

[0118] Step S702: cutting the video in MP4 format according to the error information, the context information and the configuration parameters.

[0119] In the above embodiment, the associated context information is determined based on the error information, and the corresponding relationship between the operation log and the video file is determined based on the timestamp information, and then the MP4 format video is cut according to the configuration parameters to generate a video corresponding to the error information and the context information.

[0120] Figure 8 Flowchart (VII) of the method for generating a log video of a user interaction interface in an exemplary embodiment of the present disclosure.

[0121] In an exemplary embodiment of the present disclosure, Figure 8 As shown, cutting the video in MP4 format according to the error information, the context information and the configuration parameters includes:

[0122] Step S802: parsing the video in the MP4 format.

[0123] Step S804: determine whether the MP4 format video has a mask according to the analysis result.

[0124] Step S806: If the video in the MP4 format has a mask, the MP4 format is saved in a video file with a mask.

[0125] Step S808: cutting the video in MP4 format according to the configuration parameters.

[0126] Step S810: If it is determined that the videos after the cutting process do not have masks, the video files after the cutting process are merged.

[0127] Step S812: If it is determined that the video after the cutting process has a mask, the concatenation result of the mask is output.

[0128] Fig. 9 It is an overall flow chart of a log video generation scheme of a user interaction interface in an exemplary embodiment of the present disclosure.

[0129] like Fig. 9 As shown, the steps of the log video generation scheme of the user interaction interface in the exemplary embodiment of the present disclosure include:

[0130] Step S902, setting a timestamp: setting a timestamp before the automation is executed, and the subsequent video recording and log file recording will use the timestamp as the starting time point of 00:00:00.

[0131] Step S904, UI automation execution: When UI automation is executed, the video recording component is called to generate a recorded video MP4, and the log is recorded in the form of SRT to generate an SRT file.

[0132] Step S906, step S908 and step S914: When the UI automation is executed, when an assertion is encountered, a judgment check is performed. If it passes, the execution continues. If it fails, the current timestamp is recorded.

[0133] Step S910, calling the video recording component.

[0134] Step S912, log recording.

[0135] Step S916, generate an SRT file.

[0136] Step S918, filter Error and Warning according to timestamp: take the timestamp recorded when the checkpoint fails as the input parameter, combine with the SRT file, all the Error and Warning that appear, and filter out the upper and lower 1min logs of the Error and Warning level logs to generate a new SRT file.

[0137] Step S920, video clipping: using the generated SRT file and the generated MP4 file as input parameters, calling the video cropping component, and generating a context video in which an error occurs.

[0138] Step S922, sending email report: sending the automated execution report via email with the generated video as an attachment.

[0139] Fig.10 4 is a flowchart of a video recording process in a log video generation solution of a user interaction interface in an exemplary embodiment of the present disclosure.

[0140] like Fig.10 As shown, the video recording process in the log video generation scheme of the user interaction interface in the exemplary embodiment of the present disclosure includes:

[0141] Step S1002, obtaining the current screen: calling the grab function of the Pillow library to grab the current screen.

[0142] Step S1004, obtaining the screen size: obtaining the length and width of the current screen.

[0143] Step S1006, setting the file suffix to MP4: calling the opencv-python library VideoWriter_fourcc function to set the video suffix to MP4.

[0144] Step S1008, set the file name, encoder, frame rate, and video width and height: call the opencv-python library VideoWriter function to set the video encoder, frame rate, and video length and width.

[0145] Step S1010, obtaining the current timestamp as the start time: intercepting the timestamp of the current timestamp as the start time of the video 00:00:00.

[0146] Step S1012, cyclically capture the current screen image: cyclically capture the screen RGB image through the Pillow library grab function.

[0147] Step S1014, converting the RGB image to BGR format: converting the RGB image to the BGR mode of opencv through the cvtColor function of the opencv-python library.

[0148] Step S1016, writing video: writing the video into the input file MP4.

[0149] Fig.11 The present invention is a flowchart of obtaining a subtitle file in a log video generation solution of a user interaction interface in an exemplary embodiment of the present invention.

[0150] like Fig.11 As shown, obtaining a subtitle file in the log video generation scheme of the user interaction interface in the exemplary embodiment of the present disclosure includes the following steps:

[0151] Step S1102, obtaining the current timestamp as the start time: intercepting the timestamp of the current timestamp as the start time of the log.

[0152] Step S1104, record five levels of logs: record the start time and end time of the current scenario execution, and record five levels of logs.

[0153] Step S1106, outputting the SRT file: arranging the log format and outputting the SRT log file.

[0154] Fig.12 Schematic diagram of a video cutting process of a log video generation solution of a user interaction interface in an exemplary embodiment of the present disclosure.

[0155] like Fig.12 As shown, the video cutting process of the log video generation solution of the user interaction interface in the exemplary embodiment of the present disclosure includes the following steps:

[0156] Step S1202, read the SRT text.

[0157] Step S1204, read the MP4 video file.

[0158] Step S1206, reading the BGR format file.

[0159] Step S1208, establishing an association relationship between the start time and the end time of the execution of the SRT text statement and the video time through the timestamp.

[0160] Step S1210, perform video mask analysis: If the video has a mask, call the write_videofile method of the MoviePy library to save it to a video file with a mask.

[0161] Step S1212, set video resolution: set desired_height, desired_width to let ffmpeg resize the frame before returning. If any of the parameters is empty, the frame is resized by maintaining the existing aspect ratio.

[0162] Step S1214, adjust the video size: adjust the video size by the bicubic algorithm by default, or use bilinear or fast_bilinear.

[0163] Step S1216, set the pixel format of the video to be read: if not specified, "rgb24" will be used as the default format, unless "has_mask" is set to "True", then "rgba" will be used.

[0164] Step S1218, video cut: A cut of the video will be generated, which only outputs the frames of consecutive cuts, and if they are of different sizes, no correction is performed. If all cuts have no mask, the generated cut has no mask, otherwise the mask is the concatenation of the masks.

[0165] Step S1220, video merging: cutting and merging related videos.

[0166] Corresponding to the above method embodiment, the present disclosure also provides a log video generating device of a user interaction interface, which can be used to execute the above method embodiment.

[0167] Fig.13 It is a block diagram of a log video generating device of a user interaction interface in an exemplary embodiment of the present disclosure.

[0168] refer to Fig.13 , the log video generating device 1300 of the user interaction interface may include:

[0169] The recording module 1302 is configured to perform video recording of the user interaction interface.

[0170] The recording module 1304 is configured to record the operation log of the automatic execution of the user interaction interface during the video recording process and export it as a subtitle file.

[0171] The determination module 1306 is configured to determine context information associated with the error information in the subtitle file.

[0172] The cutting module 1308 is configured to cut the recorded video according to the context information to generate the log video.

[0173] In an exemplary embodiment of the present disclosure, the recording module 1302 is further configured to:

[0174] Before recording the video of the user interaction interface, call the grab function to grab the user interaction interface to be recorded to determine the size information of the user interaction interface;

[0175] Call the VideoWriter_fourcc function to set the suffix of the video file to be recorded, and the suffix is ​​used to set the video file to MP4 format;

[0176] Call the VideoWriter function to set the configuration parameters of the video recording, where the configuration parameters include at least one of the encoder, frame rate, and video length and width.

[0177] In an exemplary embodiment of the present disclosure, the recording module 1302 is further configured to:

[0178] Recording a timestamp corresponding to an initial moment of recording the user interaction interface as a first timestamp;

[0179] Call the grab function to cyclically grab the RGB image of the user interaction interface;

[0180] The RGB image is converted into an image in BGR mode of an opencv application program through a cvtColor function, and the opencv application program can perform image analysis processing on the image in BGR mode;

[0181] The video recording of the user interaction interface is stored in MP4 format, and the attribute information of the MP4 format includes the first timestamp.

[0182] In an exemplary embodiment of the present disclosure, the recording module 1304 is further configured to:

[0183] Determine a timestamp of an initial moment of the automation execution, recorded as a second timestamp;

[0184] Recording the start time and end time of recording the user interaction interface;

[0185] Recording the operation log based on the start time and the end time;

[0186] Determine a preset level to which the level of the operation log belongs, wherein the preset level includes a DEBUG level, an INFO level, a WARN level, an ERROR level, and a FATAL level;

[0187] After determining the level of the operation log, the operation log is exported as the subtitle file.

[0188] In an exemplary embodiment of the present disclosure, the determination module 1306 is further configured to:

[0189] Establishing an association relationship between the subtitle file and the video in MP4 format according to the first timestamp and the second timestamp;

[0190] Determine the WARN level running log and the ERROR level running log in the subtitle file;

[0191] The context information of the error message is determined according to the WARN-level operation log and the ERROR-level operation log.

[0192] In an exemplary embodiment of the present disclosure, the cutting module 1308 is further configured to:

[0193] The video in MP4 format is cut according to the error information, the context information and the configuration parameters.

[0194] In an exemplary embodiment of the present disclosure, the cutting module 1308 is further configured to:

[0195] Parsing the video in MP4 format;

[0196] Determine whether the video in MP4 format has a mask according to the parsing result;

[0197] If the video in the MP4 format has a mask, the MP4 format is saved in a video file with a mask;

[0198] Performing a cutting process on the video in the MP4 format according to the configuration parameters;

[0199] If it is determined that the videos after the cutting process do not have masks, then merging the video files after the cutting process;

[0200] If it is determined that the video after the cutting process has a mask, the concatenation result of the mask is output.

[0201] Since the functions of the device 1300 have been described in detail in the corresponding method embodiments, the present disclosure will not elaborate on them here.

[0202] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0203] In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0204] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".

[0205] Refer to the following Fig.14 14 to 15. The electronic device 1400 according to this embodiment of the present invention is described. Fig.14 The electronic device 1400 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0206] like Fig.14 As shown, the electronic device 1400 is in the form of a general computing device. The components of the electronic device 1400 may include but are not limited to: at least one processing unit 1410, at least one storage unit 1420, and a bus 1430 connecting different system components (including the storage unit 1420 and the processing unit 1410).

[0207] The storage unit stores program codes, which can be executed by the processing unit 1410, so that the processing unit 1410 performs the steps according to various exemplary embodiments of the present invention described in the above “Exemplary Method” section of this specification. For example, the processing unit 1410 can perform the method shown in the embodiment of the present disclosure.

[0208] The storage unit 1420 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 14201 and / or a cache storage unit 14202 , and may further include a read-only storage unit (ROM) 14203 .

[0209] The storage unit 1420 may also include a program / utility 14204 having a set (at least one) of program modules 14205, such program modules 14205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0210] Bus 1430 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0211] The electronic device 1400 may also communicate with one or more external devices 1440 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1400, and / or communicate with any device that enables the electronic device 1400 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1450. Furthermore, the electronic device 1400 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 1460. As shown, the network adapter 1460 communicates with other modules of the electronic device 1400 via a bus 1430. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1400, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0212] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0213] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of the present specification.

[0214] The program product for implementing the above method according to an embodiment of the present invention can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto, and in this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0215] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0216] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0217] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0218] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0219] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0220] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are to be considered exemplary only, and the true scope and concept of the present disclosure are indicated by the claims.

Claims

1. A log video generation method for a user interaction interface, characterized in that: include: Record the user interface video; Recording the running log of the automatic execution of the user interaction interface during the video recording process and exporting it as a subtitle file; Determining context information associated with the error message in the subtitle file; The recorded video is cut according to the context information to generate the log video.

2. The log video generation method of the user interaction interface according to claim 1, characterized in that: Before recording the video of the user interaction interface, it also includes: Calling the grab function to grab the user interaction interface to be recorded, so as to determine the size information of the user interaction interface; Call the VideoWriter_fourcc function to set the suffix of the video file to be recorded, and the suffix is ​​used to set the video file to MP4 format; Call the VideoWriter function to set the configuration parameters of the video recording, where the configuration parameters include at least one of the encoder, frame rate, and video length and width.

3. The log video generation method of the user interaction interface according to claim 2, characterized in that: Video recording of the user interaction interface includes: Recording a timestamp corresponding to an initial moment of recording the user interaction interface as a first timestamp; Call the grab function to cyclically grab the RGB image of the user interaction interface; The RGB image is converted into an image in BGR mode of an opencv application program through a cvtColor function, and the opencv application program can perform image analysis processing on the image in BGR mode; The video recording of the user interaction interface is stored in MP4 format, and the attribute information of the MP4 format includes the first timestamp.

4. The log video generation method of the user interaction interface according to claim 3, characterized in that: Recording the operation log of the automatic execution of the user interaction interface during the video recording process and exporting it as a subtitle file includes: Determine a timestamp of an initial moment of the automation execution, recorded as a second timestamp; Recording the start time and end time of recording the user interaction interface; Recording the operation log based on the start time and the end time; Determine a preset level to which the level of the operation log belongs, wherein the preset level includes a DEBUG level, an INFO level, a WARN level, an ERROR level, and a FATAL level; After determining the level of the operation log, the operation log is exported as the subtitle file.

5. The log video generation method of the user interaction interface according to claim 4, characterized in that: Determining the context information associated with the error message in the subtitle file includes: Establishing an association relationship between the subtitle file and the video in MP4 format according to the first timestamp and the second timestamp; Determine the WARN level running log and the ERROR level running log in the subtitle file; The context information of the error message is determined according to the WARN-level operation log and the ERROR-level operation log.

6. The log video generation method of the user interaction interface according to claim 5, characterized in that: Cutting the recorded video according to the context information to generate the log video includes: The video in MP4 format is cut according to the error information, the context information and the configuration parameters.

7. The log video generation method of the user interaction interface according to claim 6, characterized in that: Cutting the video in MP4 format according to the error information, the context information and the configuration parameters includes: Parsing the video in MP4 format; Determine whether the video in MP4 format has a mask according to the parsing result; If the video in the MP4 format has a mask, the MP4 format is saved in a video file with a mask; Performing a cutting process on the video in the MP4 format according to the configuration parameters; If it is determined that the videos after the cutting process do not have masks, then merging the video files after the cutting process; If it is determined that the video after the cutting process has a mask, the concatenation result of the mask is output.

8. A log video generating device of a user interaction interface, characterized in that: include: A recording module, configured to record a video of the user interaction interface; A recording module, configured to record a running log of the automatic execution of the user interaction interface during the video recording process, and export it as a subtitle file; A determination module, configured to determine context information associated with the error message in the subtitle file; The cutting module is configured to cut the recorded video according to the context information to generate the log video.

9. An electronic device, characterized in that: include: Memory; as well as A processor coupled to the memory, the processor being configured to execute the log video generating method of the user interaction interface according to any one of claims 1 to 7 based on instructions stored in the memory.

10. A computer-readable storage medium having a program stored thereon, wherein when the program is executed by a processor, the log video generating method of the user interaction interface according to any one of claims 1 to 7 is implemented.

Images