Data processing method and electronic device
By recording screens and labeling operation during mobile phone function testing, video clips are provided for developers to analyze the causes of abnormalities, the problem of difficulty in recording and analyzing test operations in the prior art is solved, and the abnormal positioning efficiency is improved.
Patent Information
- Application Number
- CN202311375083.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-20
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2043-10-20
AI Technical Summary
During the mobile phone function testing process, it is difficult for the existing technology to effectively record and analyze the operation of testers, which makes it difficult to locate the causes of functional abnormalities, affecting developers to quickly eliminate abnormalities.
By recording the screen on the electronic device, the parameters of the operation are obtained, such as input time, position and type, and the control gestures and description information are marked in the video, providing video clips for developers to analyze the causes of the abnormality.
It improves the efficiency and accuracy of developers' analysis of functional abnormalities, reduces the difficulty of abnormal positioning, reduces the length of videos that users need to watch, and saves time and energy.
Smart Images

Figure CN118444865B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of terminal equipment, and in particular to a data processing method and electronic equipment. Background Art
[0002] With the rapid development of technology, more and more mobile phones are on the market. After developers develop a mobile phone and before the developed mobile phone is sold to the outside, the developers often need to submit the mobile phone to testers for functional testing of the mobile phone. The testers can manually test the functions of the mobile phone. When the test results indicate that the functions of the mobile phone are normal, the mobile phone can be sold to the outside. Summary of the invention
[0003] The present application provides a data processing method and an electronic device. During the function test of the first electronic device, the screen of the first electronic device can be recorded. If the tester inputs a control operation on the first electronic device during the function test of the first electronic device, the control parameters of the control operation are obtained. The control parameters include at least the input time when the control operation is input, the control position of the control operation, and the control type of the control operation. The image frame corresponding to the input time is determined in the recorded video, and the control gesture of the control operation is drawn at the control position in the image frame. The description information of the control operation is marked next to the control gesture of the control operation, and the description information of the control operation can be obtained according to the control parameters of the control operation. In this way, the developer can obtain the real situation of the abnormality of the function of the first electronic device based on the recorded video, which is conducive to the developer to analyze the cause of the abnormal function of the first electronic device, and then to eliminate the abnormality.
[0004] In a first aspect, the present application provides a data processing method, which is applied to a data processing system, wherein the data processing system includes a first electronic device and a second electronic device, wherein the first electronic device projects a screen to the second electronic device through a communication connection, and the second electronic device records a video of the first electronic device projecting the screen to the second electronic device;
[0005] The method comprises:
[0006] During the screen projection of the first electronic device to the second electronic device, the first electronic device obtains a manipulation parameter of the manipulation operation in response to a manipulation operation on the first electronic device; the manipulation parameter at least includes an input time when the manipulation operation is input, a manipulation position of the manipulation operation, and a manipulation type of the manipulation operation;
[0007] The first electronic device sends the control parameter to the second electronic device;
[0008] The second electronic device receives the control parameter;
[0009] The second electronic device searches for an image frame corresponding to the input moment in the video recorded by the second electronic device; and at least marks a manipulation position and a manipulation type of the manipulation operation in the image frame corresponding to the input moment.
[0010] During the functional test of the first electronic device, the screen of the first electronic device can be recorded, and if the tester inputs a manipulation operation on the first electronic device during the functional test of the first electronic device, the manipulation parameters of the manipulation operation can be obtained, for example, the input time when the manipulation operation is input, the manipulation position of the manipulation operation, the manipulation type of the manipulation operation, the manipulation object targeted by the manipulation operation, the manipulation direction of the manipulation operation, etc. Then, the image frame corresponding to the input time is determined in the recorded video, the manipulation gesture of the manipulation operation is drawn at the manipulation position in the image frame, and the description information of the manipulation operation is marked next to the manipulation gesture of the manipulation operation, and the description information of the manipulation operation can be obtained according to the manipulation parameters of the manipulation operation.
[0011] Sometimes, the content in the interface of the first electronic device changes during the functional test of the first electronic device. Thus, during the functional test of the first electronic device, the screen of the first electronic device can be recorded, and the recorded video can reflect the display of the interface of the first electronic device during the functional test of the first electronic device. The display of the interface of the first electronic device during the functional test of the first electronic device reflected in the recorded video is real.
[0012] Secondly, during the function test of the first electronic device, if the function of the first electronic device is abnormal, sometimes the abnormal situation will be reflected on the interface of the first electronic device. In this way, the recorded video can be fed back to the developer. In this way, the developer can obtain the real situation of the abnormality of the function of the first electronic device based on the recorded video. This is helpful for the developer to analyze the cause of the abnormality of the function of the first electronic device, and then help to eliminate the abnormality.
[0013] Furthermore, during the functional testing of the first electronic device, the tester sometimes inputs some control operations on the first electronic device, such as a single-click operation of a virtual button in the interface of the first electronic device, a long-press operation of a virtual button in the interface of the first electronic device, and a sliding operation input on the interface of the first electronic device, etc. Sometimes, these input control operations trigger the function of the first electronic device to malfunction.
[0014] In this way, the recorded video can not only reflect the display status of the interface of the first electronic device when the function of the first electronic device is abnormal, but also reflect the control status of the control operations input to the first electronic device during the function test of the first electronic device. In this way, the developer can know which control operations triggered the abnormality of the function of the first electronic device based on the recorded video. In this way, the difficulty for the developer to analyze the cause of the functional abnormality of the first electronic device with the help of the recorded video can be reduced, which is more conducive to eliminating the abnormality.
[0015] In an optional implementation, the method further includes:
[0016] The second electronic device captures a video segment including the marked image frame in the recorded video in response to the manipulation operation on the second electronic device.
[0017] Through this implementation method, a video clip can be provided to the user instead of the entire recorded video. The content of the entire recorded video other than the video clip is often unrelated to the abnormality of the function of the first electronic device. In this way, the length of the video that the user needs to watch can be reduced, the user's viewing burden can be reduced, the user's time can be saved, and the efficiency of eliminating abnormalities can be improved.
[0018] In an optional implementation, the second electronic device intercepts a video segment including the marked image frame in a recorded video in response to a manipulation operation on the second electronic device, including:
[0019] In response to a manipulation operation on the second electronic device, the second electronic device captures a video segment at the end of the recorded video with a preset duration, and the video segment at the end of the recorded video with a preset duration includes the marked image frame.
[0020] Generally, when a user is operating the first electronic device, if the user finds that the function of the first electronic device is abnormal, the user often stops operating the first electronic device, that is, stops inputting operating operations in the first electronic device, and controls the second electronic device to intercept a video clip at the end of the recorded video with a preset duration, so that the intercepted video clip includes the video of a period of time before and after the function of the first electronic device is abnormal, that is, the intercepted video can reflect the display of the interface of the first electronic device within a period of time before and after the function of the first electronic device is abnormal. In this way, the method of intercepting the video clip including the marked image frame in this implementation is simple and efficient.
[0021] In an optional implementation, the second electronic device marks at least the manipulation position and the manipulation type of the manipulation operation in the image frame corresponding to the input moment, including:
[0022] The second electronic device draws a manipulation gesture at the manipulation position of the manipulation operation in the image frame corresponding to the input moment, and at least marks the manipulation type of the manipulation operation next to the manipulation gesture.
[0023] This implementation method can visually restore the scene of the manipulation operation (such as a click operation or a sliding operation, etc.) input by the user on the first electronic device by drawing the manipulation gesture of the manipulation operation and marking the manipulation type of the manipulation operation, and the restoration is highly authentic.
[0024] In an optional implementation, the second electronic device draws a manipulation gesture at the manipulation position of the manipulation operation in the image frame corresponding to the input moment, including:
[0025] The second electronic device selects a manipulation gesture that matches the manipulation type of the manipulation operation from a plurality of preset manipulation gestures;
[0026] The second electronic device draws the adapted manipulation gesture at the manipulation position.
[0027] Among them, the control gestures adapted to the control types of different control operations are different. This implementation method can further improve the authenticity of the scene of restoring the control operation input by the user on the first electronic device by drawing the control gestures adapted to the control type of the control operation input by the user on the first electronic device.
[0028] In an optional implementation, the method further includes:
[0029] The second electronic device acquires a plurality of image frames to be inserted; the image frames to be inserted are marked with at least a manipulation position of the manipulation operation and a manipulation type of the manipulation operation;
[0030] The second electronic device inserts the plurality of image frames to be inserted between the marked image frame and the next image frame in the recorded video;
[0031] The next image frame is located after the marked image frame and is adjacent to the marked image frame.
[0032] Through this implementation method, the control gestures of the control operations are respectively drawn and the descriptive information of the control operations are marked at the control positions in multiple image frames. In this way, when the switching frequency of image frames is fast during video playing, the user's naked eyes can also perceive that the control positions in multiple image frames respectively have the control gestures of the control operations and the descriptive information of the control operations, and then can analyze what triggers the functional abnormality of the first electronic device (triggered by which control operation input at which position on which interface).
[0033] In an optional implementation, the manipulation operation on the first electronic device includes a manipulation operation in which the manipulation position does not change;
[0034] The second electronic device acquires a plurality of image frames to be inserted, including:
[0035] The second electronic device generates the multiple image frames to be inserted according to the marked image frames.
[0036] This implementation supports acquiring multiple image frames to be inserted when the manipulation operation of the first electronic device includes a manipulation operation in which the manipulation position does not change, and can restore the real scene of the manipulation operation on the first electronic device.
[0037] In an optional implementation, the second electronic device generates a plurality of image frames to be inserted according to the marked image frames, including:
[0038] The second electronic device makes a plurality of copies of the marked image frame;
[0039] The second electronic device obtains the multiple image frames to be inserted according to the multiple copies of the marked image frames.
[0040] This implementation method obtains multiple image frames to be inserted by copying multiple copies of the marked image frame, so as to improve the speed and convenience of obtaining the multiple image frames to be inserted.
[0041] In an optional implementation, the manipulation operation on the first electronic device is a manipulation operation in which the manipulation position gradually changes; the manipulation parameters of the manipulation operation include multiple manipulation positions of the manipulation operation, and the multiple manipulation positions have a sequence;
[0042] The second electronic device acquires a plurality of image frames to be inserted, including:
[0043] The second electronic device generates a plurality of image frames to be annotated according to the image frame corresponding to the input moment, and the plurality of image frames to be annotated have a sequence;
[0044] The second electronic device marks the corresponding manipulation position and the manipulation type of the manipulation operation in each image frame to be marked according to the sequence between the multiple manipulation positions and the sequence between the multiple image frames to be marked, so as to obtain the multiple image frames to be inserted;
[0045] The earlier the sequence of the marked manipulation positions is, the earlier the sequence of the marked image frames to be marked is; and the later the sequence of the marked manipulation positions is, the later the sequence of the marked image frames to be marked is.
[0046] This implementation supports acquiring multiple image frames to be inserted when the manipulation operation of the first electronic device is a manipulation operation in which the manipulation position gradually changes, and can restore the real scene of the manipulation operation on the first electronic device.
[0047] In an optional implementation, the second electronic device generates a plurality of image frames to be annotated according to the image frame corresponding to the input time, including:
[0048] The second electronic device makes a plurality of copies of the image frame corresponding to the input time;
[0049] The second electronic device obtains the multiple image frames to be marked based on the multiple copies of the image frames corresponding to the input moments.
[0050] This implementation method obtains multiple image frames to be annotated by making multiple copies of the image frame corresponding to the input moment, so as to improve the speed and convenience of obtaining the multiple image frames to be annotated.
[0051] In an optional implementation, the method further includes:
[0052] The first electronic device sends the interception time of the image frame of the projected screen to the second electronic device, where the interception time includes the time when the first electronic device intercepts the image frame of the projected screen;
[0053] The second electronic device receives the interception time; and stores the identifier of the image frame of the projected screen and the interception time in the correspondence between the identifier of the image frame and the interception time;
[0054] The second electronic device searches for an image frame corresponding to the input time in the video recorded by the second electronic device, including:
[0055] The second electronic device uses the input time as the interception time, and searches for the image frame corresponding to the input time in the corresponding relationship.
[0056] This implementation method can improve the efficiency of searching for image frames corresponding to input moments.
[0057] In an optional implementation, the first electronic device includes a mobile phone; the second electronic device includes a laptop computer, a desktop computer, a tablet computer or a smart screen.
[0058] In a second aspect, the present application provides a data processing method, which is applied to a first electronic device in a data processing system, wherein the data processing system also includes a second electronic device, wherein the first electronic device projects a screen to the second electronic device through a communication connection, and the second electronic device records a video of the first electronic device projecting a screen to the second electronic device, and the method includes:
[0059] In the process of projecting the screen of the first electronic device to the second electronic device, in response to a manipulation operation on the first electronic device, a manipulation parameter of the manipulation operation is acquired; the manipulation parameter at least includes an input time when the manipulation operation is input, a manipulation position of the manipulation operation, and a manipulation type of the manipulation operation;
[0060] The control parameters are sent to the second electronic device so that the second electronic device searches for an image frame corresponding to the input moment in the video recorded by the second electronic device; in the image frame corresponding to the input moment, at least a control position of the control operation and a control type of the control operation are marked.
[0061] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.
[0062] In a third aspect, the present application provides a data processing method, which is applied to a second electronic device in a data processing system, wherein the data processing system also includes a first electronic device, wherein the first electronic device projects a screen to the second electronic device through a communication connection, and the second electronic device records a video of the first electronic device projecting a screen to the second electronic device, and the method includes:
[0063] receiving a manipulation parameter of a manipulation operation sent by the first electronic device to the second electronic device; the manipulation parameter at least includes an input time when the manipulation operation is input, a manipulation position of the manipulation operation, and a manipulation type of the manipulation operation; the manipulation parameter is acquired by the first electronic device in response to the manipulation operation on the first electronic device during the process of projecting the screen of the first electronic device to the second electronic device;
[0064] Searching for an image frame corresponding to the input moment in the video recorded by the second electronic device;
[0065] In the image frame corresponding to the input moment, at least the manipulation position of the manipulation operation and the manipulation type of the manipulation operation are marked.
[0066] The third aspect and any implementation of the third aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the third aspect and any implementation of the third aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.
[0067] In a fourth aspect, the present application provides a first electronic device, including:
[0068] One or more processors, memory;
[0069] and one or more computer programs, wherein the one or more computer programs are stored in the memory, and when the computer programs are executed by the one or more processors, the first electronic device performs the following steps:
[0070] In a process in which a first electronic device projects a screen onto a second electronic device and the second electronic device records a video of the projected screen, in response to a manipulation operation on the first electronic device, a manipulation parameter of the manipulation operation is acquired; the manipulation parameter at least includes an input time when the manipulation operation is input, a manipulation position of the manipulation operation, and a manipulation type of the manipulation operation;
[0071] The control parameters are sent to the second electronic device so that the second electronic device searches for an image frame corresponding to the input moment in the video recorded by the second electronic device; in the image frame corresponding to the input moment, at least a control position of the control operation and a control type of the control operation are marked.
[0072] In an optional implementation, when the program instruction is executed by the processor, the first electronic device executes the following steps:
[0073] Sending the capture time of the image frame of the projected screen to the second electronic device, where the capture time includes the time when the first electronic device captures the image frame of the projected screen;
[0074] So that the second electronic device receives the interception time; in the correspondence between the identifier of the image frame and the interception time, the identifier of the projected image frame and the interception time are stored; and when searching for the image frame corresponding to the input time in the video recorded by the second electronic device, the input time is used as the interception time, and the image frame corresponding to the input time is searched in the correspondence.
[0075] The fourth aspect and any implementation of the fourth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fourth aspect and any implementation of the fourth aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, which will not be repeated here.
[0076] In a fifth aspect, the present application provides a second electronic device, including:
[0077] One or more processors, memory;
[0078] and one or more computer programs, wherein the one or more computer programs are stored in the memory, and when the computer programs are executed by the one or more processors, the second electronic device performs the following steps:
[0079] In the process of a first electronic device projecting a screen to a second electronic device and the second electronic device recording a video of the projected screen, receiving a manipulation parameter of a manipulation operation sent by the first electronic device to the second electronic device; the manipulation parameter at least includes an input time when the manipulation operation is input, a manipulation position of the manipulation operation, and a manipulation type of the manipulation operation; the manipulation parameter is obtained by the first electronic device in response to the manipulation operation on the first electronic device in the process of the first electronic device sending a projected image frame to the second electronic device;
[0080] Searching for an image frame corresponding to the input moment in the video recorded by the second electronic device;
[0081] In the image frame corresponding to the input moment, at least the manipulation position of the manipulation operation and the manipulation type of the manipulation operation are marked.
[0082] In an optional implementation, when the program instruction is executed by the processor, the second electronic device executes the following steps:
[0083] In response to a manipulation operation on the second electronic device, a video segment including the marked image frames is captured from the recorded video.
[0084] In an optional implementation, when the program instruction is executed by the processor, the second electronic device executes the following steps:
[0085] In response to the manipulation of the second electronic device, in the recorded video, a video segment at the end with a preset duration is captured, and the video segment at the end with a preset duration includes the marked image frame.
[0086] In an optional implementation, when the program instruction is executed by the processor, the second electronic device executes the following steps:
[0087] A manipulation gesture is drawn at the manipulation position of the manipulation operation in the image frame corresponding to the input moment, and at least the manipulation type of the manipulation operation is marked next to the manipulation gesture.
[0088] In an optional implementation, when the program instruction is executed by the processor, the second electronic device executes the following steps:
[0089] Selecting a manipulation gesture that matches the manipulation type of the manipulation operation from among a plurality of preset manipulation gestures;
[0090] The adapted manipulation gesture is drawn at the manipulation position.
[0091] In an optional implementation, when the program instruction is executed by the processor, the second electronic device executes the following steps:
[0092] Acquire a plurality of image frames to be inserted; the image frames to be inserted are at least marked with a manipulation position of the manipulation operation and a manipulation type of the manipulation operation;
[0093] Inserting the plurality of image frames to be inserted between the marked image frame and the next image frame in the recorded video;
[0094] The next image frame is located after the marked image frame and is adjacent to the marked image frame.
[0095] In an optional implementation, when the program instruction is executed by the processor, the second electronic device executes the following steps:
[0096] The plurality of image frames to be inserted are generated according to the marked image frames, and the manipulation operation on the first electronic device includes a manipulation operation in which the manipulation position does not change.
[0097] In an optional implementation, when the program instruction is executed by the processor, the second electronic device executes the following steps:
[0098] Copying a plurality of copies of the labeled image frame;
[0099] The multiple image frames to be inserted are obtained according to the multiple copies of the marked image frames.
[0100] In an optional implementation, when the program instruction is executed by the processor, the second electronic device executes the following steps:
[0101] Generate a plurality of image frames to be annotated according to the image frame corresponding to the input time, wherein the plurality of image frames to be annotated have a sequence;
[0102] According to the sequence between the multiple control positions and the sequence between the multiple image frames to be annotated, the corresponding control position and the control type of the control operation are respectively annotated in each image frame to be annotated, so as to obtain the multiple image frames to be inserted; the control operation on the first electronic device is a control operation with gradually changing control positions; there are multiple control positions of the control operation in the control parameters of the control operation, and the multiple control positions have a sequence;
[0103] The earlier the sequence of the marked manipulation positions is, the earlier the sequence of the marked image frames to be marked is; and the later the sequence of the marked manipulation positions is, the later the sequence of the marked image frames to be marked is.
[0104] In an optional implementation, when the program instruction is executed by the processor, the second electronic device executes the following steps:
[0105] Copying a plurality of copies of the image frame corresponding to the input time;
[0106] The multiple image frames to be marked are obtained according to the multiple copies of the image frames corresponding to the input time.
[0107] In an optional implementation, when the program instruction is executed by the processor, the second electronic device executes the following steps:
[0108] receiving the capture time of the projected image frame sent by the first electronic device; storing the identifier of the projected image frame and the capture time in the correspondence between the identifier of the image frame and the capture time, wherein the capture time includes the time when the first electronic device captures the projected image frame;
[0109] And, taking the input time as the interception time, searching for the image frame corresponding to the input time in the corresponding relationship.
[0110] The fifth aspect and any implementation of the fifth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fifth aspect and any implementation of the fifth aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.
[0111] In a sixth aspect, the present application provides a computer storage medium comprising computer instructions. When the computer instructions are executed on a first electronic device, the first electronic device executes the steps of the method executed by the first electronic device as shown in the first aspect, any one of the implementations in the first aspect, the second aspect, and any one of the implementations in the second aspect.
[0112] The sixth aspect and any implementation of the sixth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the sixth aspect and any implementation of the sixth aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.
[0113] In the seventh aspect, the present application provides a computer storage medium comprising computer instructions. When the computer instructions are executed on a second electronic device, the second electronic device executes the steps of the method executed by the second electronic device as shown in the first aspect, any one of the implementations in the first aspect, the third aspect, or any one of the implementations in the third aspect.
[0114] The seventh aspect and any implementation of the seventh aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the seventh aspect and any implementation of the seventh aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.
[0115] In an eighth aspect, the present application provides a computer program product. When the computer program product runs on a computer, the computer executes the steps of the method executed by the first electronic device shown in the first aspect, any one of the implementations of the first aspect, the second aspect, and any one of the implementations of the second aspect.
[0116] The eighth aspect and any implementation of the eighth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the eighth aspect and any implementation of the eighth aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.
[0117] In the ninth aspect, the present application provides a computer program product. When the computer program product runs on a computer, it enables the computer to execute the steps of the method executed by the second electronic device shown in the first aspect, any one of the implementations of the first aspect, the third aspect, and any one of the implementations of the third aspect.
[0118] The ninth aspect and any implementation of the ninth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the ninth aspect and any implementation of the ninth aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.
[0119] In the tenth aspect, the present application provides a chip comprising one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from a memory of a first electronic device and send the signal to the processor, the signal comprising computer instructions stored in the memory; when the processor executes the computer instructions, the first electronic device executes the steps of the method executed by the first electronic device as shown in the first aspect, any one of the implementations in the first aspect, the second aspect, and any one of the implementations in the second aspect.
[0120] The tenth aspect and any implementation of the tenth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the tenth aspect and any implementation of the tenth aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.
[0121] In the eleventh aspect, the present application provides a chip comprising one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from a memory of a second electronic device and send the signal to the processor, the signal comprising computer instructions stored in the memory; when the processor executes the computer instructions, the second electronic device executes the steps of the method executed by the second electronic device as shown in the first aspect, any one of the implementations of the first aspect, the third aspect, and any one of the implementations of the third aspect.
[0122] The eleventh aspect and any implementation of the eleventh aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the eleventh aspect and any implementation of the eleventh aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.
[0123] In a twelfth aspect, the present application provides a data processing system, which includes the first electronic device shown in the first and second aspects above and the second electronic device shown in the first and third aspects above. BRIEF DESCRIPTION OF THE DRAWINGS
[0124] Figure 1 is a schematic diagram of an exemplary screen projection scenario;
[0125] Figure 2 is a schematic diagram of an exemplary screen projection scenario;
[0126] Figure 3 is a schematic diagram of an exemplary screen projection scenario;
[0127] Figure 4 is a schematic diagram of an exemplary screen projection scenario;
[0128] Figure 5 is a schematic diagram of the structure of an electronic device shown as an example;
[0129] Figure 6 A schematic diagram of the software structure of an electronic device is shown as an example;
[0130] Figure 7 The figure is a schematic diagram showing an exemplary screen projection scenario between a mobile phone and a laptop computer;
[0131] Figure 8 A schematic diagram showing an exemplary method of establishing a Wi-Fi connection between a mobile phone and a laptop computer;
[0132] Fig. 9 The figure is a schematic diagram of a process of establishing a Wi-Fi connection between a mobile phone and a laptop computer;
[0133] Fig.10 is a schematic diagram of an exemplary mobile phone interface;
[0134] Fig.11 is a schematic diagram of data transmission between a mobile phone and a laptop computer;
[0135] Fig.12 is a schematic diagram of an interface of a laptop computer shown as an example;
[0136] Fig.13 is a schematic diagram of an interface of a laptop computer shown as an example;
[0137] Fig.14 This is a schematic diagram of an interface of a screen projection recording application displayed in an interface of a laptop computer as an example;
[0138] Fig.15 This is a schematic diagram of an interface of a screen projection recording application displayed in an interface of a laptop computer as an example;
[0139] Fig.16 This is a schematic diagram of an interface of a screen projection recording application displayed in an interface of a laptop computer as an example;
[0140] Fig.17 is a schematic diagram of an interface of a laptop computer shown as an example;
[0141] Fig.18 This is a schematic diagram of an interface of a screen projection recording application displayed in an interface of a laptop computer as an example;
[0142] Fig.19 This is a schematic diagram of an interface of a screen projection recording application displayed in an interface of a laptop computer as an example;
[0143] Fig. 20 This is a schematic diagram of an interface of a screen projection recording application displayed in an interface of a laptop computer as an example;
[0144] Fig.21 is a distribution diagram of image frames in a recorded video shown as an example;
[0145] Fig. 22 The figure is a schematic diagram showing an exemplary method of capturing a segment of a video from a recorded video;
[0146] Fig.23 The distribution diagram of image frames in a video clip is shown as an example;
[0147] Fig.24 The figure is a schematic diagram showing an exemplary method of labeling image frames in a captured video;
[0148] Fig.25 The figure is a schematic diagram showing, for example, that image frames in a captured video have been labeled;
[0149] Fig.26 The figure is a schematic diagram showing an exemplary method of inserting an image frame into a captured video;
[0150] Fig. 27 The distribution diagram of image frames in a video with image frames inserted is shown as an example;
[0151] Fig.28 is a schematic diagram of an image frame that has been labeled as an example;
[0152] Fig.29 is a schematic diagram of an image frame that has been labeled as an example;
[0153] Fig.30 is a schematic diagram of an image frame that has been labeled as an example;
[0154] Fig.31 is a schematic diagram of an image frame that has been labeled as an example;
[0155] Fig.32 is a schematic diagram of an image frame that has been labeled as an example;
[0156] Fig.33A flowchart of a data processing method is shown as an example;
[0157] Fig.34 A flowchart of a data processing method is shown as an example;
[0158] Fig.35 The figure is a schematic diagram of the structure of a device shown as an example. DETAILED DESCRIPTION
[0159] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0160] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.
[0161] The terms "first" and "second" in the description and claims of the embodiments of the present application are used to distinguish different objects rather than to describe a specific order of objects. For example, a first target object and a second target object are used to distinguish different target objects rather than to describe a specific order of target objects.
[0162] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.
[0163] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more than two. For example, multiple processing units refer to two or more processing units; multiple systems refer to two or more systems.
[0164] In the scenario of functional testing of a mobile phone, in one method, the tester can test various aspects of the mobile phone's functions in sequence. During the test, the mobile phone may have abnormal functions. After testing various aspects of the mobile phone's functions, the tester can verbally feedback the test results to the developer. For example, verbally feedback the abnormal function of the mobile phone to the developer, and then the developer can analyze the cause of the abnormal function of the mobile phone based on the tester's feedback, and then eliminate the abnormality based on the cause.
[0165] However, the process of testing various aspects of the functions of the mobile phone in sequence is time-consuming. Sometimes, the functions of the mobile phone have some abnormalities at an early time. In this way, after the functions of the mobile phone are tested in sequence in various aspects, the testers may not be able to accurately repeat the specific circumstances of these abnormalities of the functions of the mobile phone that occurred at an earlier time, which leads to errors in the abnormal circumstances of the functions of the mobile phone that appeared at an earlier time fed back to the developers, which is not conducive to the developers to analyze the causes of the abnormal functions of the mobile phone, and is not conducive to eliminating the abnormalities.
[0166] The present application provides a method that can record the screen of a mobile phone during the function test of the mobile phone, and if the tester inputs a control operation on the mobile phone during the function test of the mobile phone, the control parameters of the control operation can be obtained, such as the input time when the control operation is input, the control position of the control operation, the control type of the control operation, the control object targeted by the control operation, the control direction of the control operation, etc. Then, the image frame corresponding to the input time is determined in the recorded video, the control gesture of the control operation is drawn at the control position in the image frame, and the description information of the control operation is marked next to the control gesture of the control operation. The description information of the control operation can be obtained according to the control parameters of the control operation.
[0167] Sometimes, the content of the mobile phone interface will change during the function test of the mobile phone. In this way, during the function test of the mobile phone, the screen of the mobile phone can be recorded, and the recorded video can reflect the display of the mobile phone interface during the function test of the mobile phone. The display of the mobile phone interface during the function test of the mobile phone reflected in the recorded video is real.
[0168] Secondly, during the function test of the mobile phone, if the function of the mobile phone is abnormal, sometimes the abnormal situation will be reflected on the interface of the mobile phone. In this way, the recorded video can be fed back to the developer. In this way, the developer can get the real situation of the abnormality of the function of the mobile phone based on the recorded video, which is helpful for the developer to analyze the cause of the abnormal function of the mobile phone and then help to eliminate the abnormality.
[0169] Furthermore, during the function testing of the mobile phone, the tester sometimes inputs some control operations on the mobile phone, such as single-clicking a virtual button in the mobile phone's interface, long-pressing a virtual button in the mobile phone's interface, and sliding operations input on the mobile phone's interface, etc. Sometimes, these input control operations trigger the mobile phone's functions to malfunction.
[0170] In this way, the recorded video can not only reflect the display of the mobile phone interface when the mobile phone function fails, but also reflect the control status of the control operations input into the mobile phone during the function testing of the mobile phone. In this way, the developer can know which control operations trigger the abnormality of the mobile phone function based on the recorded video. In this way, the difficulty of developers analyzing the causes of the abnormality of the mobile phone function with the help of the recorded video can be reduced, which is more conducive to eliminating the abnormality.
[0171] Before describing the technical solution of the embodiment of the present application, the application scenario of the embodiment of the present application is first described with reference to the accompanying drawings.
[0172] The application scenario of the embodiment of the present application may include a transmitter and a receiver, and the transmitter may project a screen to the receiver. In the screen projection scenario, the transmitter and the receiver may also be referred to as P2P (Peer-to-Peer, peer-to-peer) devices.
[0173] The screen projection scenarios in the embodiments of the present application may include: screen projection scenarios between mobile phones and laptop computers, screen projection scenarios between mobile phones and smart screens (for example, smart TVs, etc.), screen projection scenarios between mobile phones and tablet computers, and screen projection scenarios between mobile phones and desktop computers, etc.
[0174] Of course, it is understandable that the screen projection scenarios in the embodiments of the present application may also include screen projection scenarios between other devices, for example, screen projection scenarios between electronic devices such as vehicle-mounted devices and wearable devices and electronic devices such as laptops, smart screens, tablets, desktop computers, etc. This application does not limit the types of electronic devices involved in the screen projection scenarios and will not give examples one by one.
[0175] For example, Figure 1 to Figure 4 Shown is a schematic diagram of an exemplary screen projection scenario.
[0176] Figure 1 Shown is a schematic diagram of an exemplary screen projection scenario between a mobile phone and a laptop computer, wherein the mobile phone can act as a sending end, the mobile phone can send relevant data of the mobile phone's interface to the laptop computer, the laptop computer can receive the relevant data of the mobile phone's interface, and display the mobile phone's interface on the laptop computer's interface based on the relevant data of the mobile phone's interface, and display the content displayed on the mobile phone's interface.
[0177] Figure 2 Shown is a schematic diagram of an exemplary screen projection scenario between a mobile phone and a smart screen, wherein the mobile phone can act as a sending end, the mobile phone can send relevant data of the mobile phone's interface to the smart screen, the smart screen can receive the relevant data of the mobile phone's interface, and display the mobile phone's interface on the smart screen's interface according to the relevant data of the mobile phone's interface, and display the content displayed on the mobile phone's interface.
[0178] Figure 3 Shown is a schematic diagram of an exemplary screen projection scenario between a mobile phone and a tablet computer, wherein the mobile phone can act as a sending end, the mobile phone can send relevant data of the mobile phone's interface to the tablet computer, the tablet computer can receive the relevant data of the mobile phone's interface, and display the mobile phone's interface on the tablet computer's interface according to the relevant data of the mobile phone's interface, and display the content displayed on the mobile phone's interface.
[0179] Figure 4 Shown is a schematic diagram of an exemplary screen projection scenario between a mobile phone and a desktop computer, wherein the mobile phone can act as a sending end, the mobile phone can send relevant data of the mobile phone interface to the desktop computer, the desktop computer can receive the relevant data of the mobile phone interface, and display the mobile phone interface on the desktop computer interface according to the relevant data of the mobile phone interface, and display the content displayed on the mobile phone interface.
[0180] Figure 5 This is a schematic diagram of the structure of an electronic device shown in an embodiment of the present application. Those skilled in the art will understand that Figure 5 The structure of the electronic device in can be applied to Figures 1 to 4 The mobile phones, laptops, smart screens, tablets, and desktop computers in the screen projection scenes are shown respectively, and it can be understood that Figure 5 The structure of the electronic device in the figure can also be applied to wearable devices, vehicle-mounted devices and other possible electronic devices not shown in the figure.
[0181] Reference Figure 5The electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.
[0182] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.
[0183] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.
[0184] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.
[0185] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.
[0186] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.
[0187] The I2C interface is a bidirectional synchronous serial bus that may include a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C buses. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example, the processor 110 may be coupled to the touch sensor 180K through the I2C interface, so that the processor 110 communicates with the touch sensor 180K through the I2C bus interface to realize the touch function of the electronic device 100.
[0188] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to achieve communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 via the I2S interface to achieve the function of answering a call through a Bluetooth headset.
[0189] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface to realize the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.
[0190] The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 through the UART interface to implement the function of playing music through a Bluetooth headset.
[0191] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193. The MIPI interface may include a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the shooting function of the electronic device 100. The processor 110 and the display screen 194 communicate via the DSI interface to implement the display function of the electronic device 100.
[0192] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193, the display 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.
[0193] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transfer data between the electronic device 100 and a peripheral device. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.
[0194] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.
[0195] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device 100 through the power management module 141.
[0196] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.
[0197] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.
[0198] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.
[0199] The mobile communication module 150 can provide a solution for wireless communications such as 2G / 3G / 4G / 5G applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.
[0200] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.
[0201] The wireless communication module 160 can provide wireless communication solutions for the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and filters it, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.
[0202] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a BeiDou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).
[0203] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.
[0204] The display screen 194 is used to display images, videos, etc. The display screen 194 may include a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.
[0205] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.
[0206] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.
[0207] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.
[0208] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.
[0209] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.
[0210] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.
[0211] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.
[0212] The internal memory 121 can be used to store computer executable program codes, which may include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.
[0213] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.
[0214] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.
[0215] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 170A.
[0216] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the human ear.
[0217] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to microphone 170C to input the sound signal into microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.
[0218] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130, or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.
[0219] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. Capacitive pressure sensors can be parallel plates that can include at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure based on the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the touch operation intensity according to the pressure sensor 180A. The electronic device 100 can also calculate the touch position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.
[0220] The gyro sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyro sensor 180B. The gyro sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyro sensor 180B detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyro sensor 180B can also be used for navigation and somatosensory game scenes.
[0221] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.
[0222] The magnetic sensor 180D may include a Hall sensor. The electronic device 100 may use the magnetic sensor 180D to detect the opening and closing of the flip leather case. In some embodiments, when the electronic device 100 is a flip phone, the electronic device 100 may detect the opening and closing of the flip cover according to the magnetic sensor 180D. Then, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, the flip cover automatic unlocking and other features may be set.
[0223] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in all directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.
[0224] The distance sensor 180F is used to measure the distance. The electronic device 100 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 can use the distance sensor 180F to measure the distance to achieve fast focusing.
[0225] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device 100 emits infrared light outward through the light emitting diode. The electronic device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100. The electronic device 100 can use the proximity light sensor 180G to detect that the user holds the electronic device 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode and pocket mode to automatically unlock and lock the screen.
[0226] The ambient light sensor 180L is used to sense the ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in a pocket to prevent accidental touches.
[0227] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.
[0228] The temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the electronic device 100 reduces the performance of a processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 heats the battery 142 to avoid abnormal shutdown of the electronic device 100 due to low temperature. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 performs a boost on the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.
[0229] The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.
[0230] The bone conduction sensor 180M can obtain a vibration signal. In some embodiments, the bone conduction sensor 180M can obtain a vibration signal of a vibrating bone block of the vocal part of the human body. The bone conduction sensor 180M can also contact the human pulse to receive a blood pressure beat signal. In some embodiments, the bone conduction sensor 180M can also be set in an earphone and combined into a bone conduction earphone. The audio module 170 can parse out a voice signal based on the vibration signal of the vibrating bone block of the vocal part obtained by the bone conduction sensor 180M to realize a voice function. The application processor can parse the heart rate information based on the blood pressure beat signal obtained by the bone conduction sensor 180M to realize a heart rate detection function.
[0231] The key 190 may include a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.
[0232] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.
[0233] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.
[0234] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 100.
[0235] Figure 6It is a schematic diagram of the software structure of the electronic device 100 according to an embodiment of the present application.
[0236] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the system framework layer, the Android runtime and system library layer, and the kernel layer.
[0237] The application layer can include applications such as camera, gallery, calendar, screen projection, WLAN, music, video, map, and Bluetooth. Figure 6 The application programs that can be included in the application layer shown in the figure are only exemplary and are not limited in this application. It is understood that the application programs that can be included in the application layer do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, compared with Figure 6 The application layer shown includes applications, and the electronic device 100 may include more or fewer applications, or the electronic device 100 may include completely different applications.
[0238] The system framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer, and can include various components and services to support developers' Android development. The system framework layer can include some predefined functions. Figure 6 As shown, the system framework layer may include a view system, a window manager, a display interface, a resource manager, a content provider, a screen projection module, and the like. The view system may include visual controls, such as controls for displaying text, controls for displaying pictures, and the like. The view system may be used to build applications. The display interface may be composed of one or more views. The window manager is used to manage window programs. The window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, capture the screen, and the like. The resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, and the like. The content provider is used to store and obtain data and make the data accessible to the application. These data may include video, images, audio, and the like. The screen projection module is used to establish a connection for screen projection based on the connection. Among them, the connection may be a Wi-Fi connection or other types of connections, and the present application does not limit this.
[0239] The Android runtime and system library layer may include a system library and an Android runtime. The system library may include multiple functional modules. For example: a browser kernel, a 3D graphics library (e.g., OpenGL ES), a font library, etc. The browser kernel is responsible for interpreting the syntax of a web page (such as an application HTML, JavaScript under a standard universal markup language) and rendering (displaying) the web page. The 3D graphics library is used to implement three-dimensional graphics drawing, image rendering, synthesis, and layer processing, etc. The font library is used to implement input of different fonts. The Android runtime may include a core library and a virtual machine. The Android runtime is responsible for scheduling and management of the Android system. The core library contains two parts: one part is the function function that the Java language needs to call, and the other part is the Android core library. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.
[0240] Understandably, Figure 6 The components included in the system framework layer, system library and runtime layer shown do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or combine some components, or split some components, or arrange the components differently.
[0241] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, sensor driver and data processing module. Among them, the data processing module is used to process the data on the Wi-Fi connection accordingly.
[0242] For example, refer to Figure 7 , Figure 7 This is a schematic diagram of an exemplary screen projection scenario between a mobile phone and a laptop computer. A 2.4GHz Wi-Fi connection (also called a 2.4GHz channel, a 2.4GHz screen projection channel, or a 2.4GHz P2P connection, etc.) can be established between the mobile phone and the laptop computer. For example, the mobile phone can exchange data with the laptop computer through the 2.4GHz Wi-Fi connection established between the mobile phone and the laptop computer. Figure 8 and Fig. 9 , which details the process of establishing a 2.4GHz Wi-Fi connection between a mobile phone and a laptop.
[0243] Of course, it is understandable that the mobile phone can also establish a Wi-Fi connection with the laptop computer at other frequency bands (such as 5 GHz, etc.). For example, the mobile phone can exchange data with the laptop computer through the Wi-Fi connection with other frequency bands (such as 5 GHz, etc.) established between the mobile phone and the laptop computer. This application does not limit the frequency band of the Wi-Fi connection established between the mobile phone and the laptop computer.
[0244] Figure 8 The figure is a schematic diagram showing an exemplary method of establishing a Wi-Fi connection between a mobile phone and a laptop computer.
[0245] Exemplarily, the process of establishing a 2.4 GHz Wi-Fi connection between a mobile phone and a laptop can be divided into three parts:
[0246] Part 1: Device discovery. Specifically, in the device discovery phase, mobile phones and laptops can search to find other P2P-enabled devices around them. For example, a mobile phone can search to find a P2P-enabled laptop. Similarly, a laptop can search to find a P2P-enabled mobile phone.
[0247] The second part is that the mobile phone and the laptop computer establish a group. Specifically, taking the mobile phone as an example, after the mobile phone finds that there is a laptop computer supporting P2P around, it can establish a P2P Group with the laptop computer and negotiate who will play the role of GO (Group Owner) and who will play the role of Client (member, also called group member).
[0248] For example, the role of GO is similar to the access point AP (Access Point) in the basic service set BSS (Basic Service Set), which can also be called a base station. The role of Client is similar to the station (STA) in the BSS. The so-called station refers to a terminal device (electronic device) with Wi-Fi communication function and connected to the wireless network, such as a mobile phone, desktop computer, smart screen, tablet computer, laptop computer, etc. The station can support multiple wireless local area networks (WLAN) standards of the 802.11 family, such as 802.11be, 802.11ax, 802.11ac, 802.11n, 802.11g, 802.11b and 802.11a.
[0249] An access point can be a terminal device (such as a mobile phone) or a network device (such as a router) with a Wi-Fi chip. For example, an access point can be a device that supports the 802.11be standard. An access point can also be a device that supports multiple WLAN standards of the 802.11 family, such as 802.11be, 802.11ax, 802.11ac, 802.11n, 802.11g, 802.11b, and 802.11a.
[0250] It should be noted that the electronic devices in the embodiments of the present application are generally terminal products that support the 802.11 series of standards. In the evolution process from 802.11a to 802.11g, 802.11n, 802.11ac to 802.11ax, the available frequency bands may include 2.4 GHz and 5 GHz. With more and more open frequency bands, the maximum channel bandwidth supported by 802.11 has expanded from 20 MHz to 40 MHz and then to 160 MHz. In 2017, the Federal Communications Commission (FCC) of the United States opened a new free frequency band of 6 GHz (5925-7125 MHz). In the 802.11ax project authorization request (PAR), the 802.11ax standard workers expanded the working range of 802.11ax devices from 2.4 GHz and 5 GHz to 2.4 GHz, 5 GHz and 6 GHz. This application only uses the example of electronic devices operating at 2.5 GHz and 5 GHz for illustration. The embodiments of this application can also be applied to screen projection scenarios between devices that support the next generation communication protocol (such as 802.11be, etc.).
[0251] The third part, the mobile phone and the laptop computer establish a 2.4GHz channel. Specifically, after the mobile phone and the laptop computer determine the GO and the Client, the GO (such as a laptop computer) can initiate a 2.4GHz channel establishment process to the Client (such as a mobile phone). After the laptop computer and the mobile phone successfully establish a 2.4GHz channel, data can be exchanged on the channel. It should be noted that the GO can be the transmitter (such as a mobile phone) or the receiver (such as a laptop computer) in the screen projection scenario, and this application does not limit it.
[0252] Combine the following Fig. 9 ,right Figure 8 The three parts shown in the figure are explained. Fig. 9 The following is a schematic diagram of a process for establishing a Wi-Fi connection between a mobile phone and a laptop computer. The process may specifically include:
[0253] S101, the mobile phone starts screen projection.
[0254] For example, Fig.10 For a schematic diagram of an exemplary interface, refer to Fig.10 (1) The display window of the mobile phone may include one or more controls, which may include but are not limited to: display interface, application icon, power icon, network icon, etc. The user can click the setting icon in the display window of the mobile phone to enter the setting interface. The mobile phone displays the setting interface in response to the user's single click operation. Fig.10 (2) The setting interface may include one or more controls, which may include but are not limited to: WLAN setting options, Bluetooth setting options, mobile network setting options, and more connection setting options. For example, the user may further click on the more connection setting option, and the mobile phone displays the more connection setting interface in response to the user's single click operation. Fig.10 (3) For example, the more connection settings interface may include one or more controls, which may include but are not limited to: NFC settings options, Honor Beam settings options, Honor Share settings options, and mobile phone screen projection settings options. For example, the user can click on the mobile phone screen projection settings option, and the mobile phone responds to the user's single click operation and displays the mobile phone screen projection settings interface. Fig.10 (4) For example, the mobile screen projection setting interface may include one or more controls, including but not limited to: unlimited screen projection setting options and a list of available devices. For example, when the unlimited screen projection setting option is not turned on, the user can manually click to turn it on. After turning it on, the mobile phone can automatically search for available P2P devices around and display the searched P2P devices in the list of available devices, for example Fig.10 The laptop shown in (4). The user can click on a P2P device shown in the list of available devices to instruct the mobile phone to project the screen to the laptop.
[0255] It should be noted that the method of triggering screen projection described in the embodiments of the present application is only an illustrative example. For example, in other embodiments, the user can also project the screen of the mobile phone through the projection option in the drop-down menu or sidebar, which is not limited in the present application.
[0256] S102, the mobile phone sends a probe request message to the laptop computer.
[0257] Still refer to Fig. 9 , exemplarily, the mobile phone sends a probe request message to the laptop computer to request communication with the laptop computer. , exemplarily, the mobile phone can send the probe request message on channels 1, 6, and 11 of the 2.4 GHz frequency band, respectively.
[0258] S103, the laptop computer sends a probe response message to the mobile phone.
[0259] Exemplarily, when the laptop receives a probe request message on any one of channels 1, 6, and 11, it may select one of the channels on which the probe request message is received, and send a probe response message to the mobile phone on the channel (eg, channel 6).
[0260] S104, the mobile phone sends a GO negotiation request message to the notebook computer.
[0261] Exemplarily, the mobile phone sends a GO negotiation request message to the laptop computer on channel 6 in response to the probe response message received from the laptop computer.
[0262] S105, the notebook computer sends a GO negotiation response message to the mobile phone.
[0263] Exemplarily, in response to receiving a GO negotiation request message from the mobile phone, the laptop sends a GO negotiation response message to the mobile phone on channel 6.
[0264] S106, the mobile phone sends a GO confirm message to the laptop computer.
[0265] It should be noted that the three frame exchanges of S104 to S106 are used to exchange information to identify the GO and the Client, and the information exchanged in the three frame exchanges may include but is not limited to: the MAC address of the GO, the Group ID, etc. In the description of the embodiments of the present application, a laptop is used as the GO and a mobile phone is used as the Client.
[0266] S107, the laptop computer sends a Beacon message to the mobile phone.
[0267] For example, after the laptop computer and the mobile phone complete the group establishment, the laptop computer as the GO end can send a Beacon message (broadcast message) (also called a Beacon frame) to establish a screen projection channel with other devices on the 2.4GHz frequency band.
[0268] For example, a notebook computer sending a Beacon message on channel 3 of 2.4 GHz is taken as an example for description.
[0269] S108, the mobile phone and the laptop computer perform link authentication association.
[0270] For example, after the mobile phone monitors the Beacon message sent by the laptop on channel 3, it performs a link authentication association process with the laptop, wherein the link authentication association process requires multiple frame exchanges to exchange information such as MAC address information, encryption method, and supported channel sets.
[0271] S109: The mobile phone and the laptop perform a four-way handshake.
[0272] For example, after 4 handshakes between the mobile phone and the laptop, a 2.4 GHz channel is successfully established on channel 3. It should be noted that Fig. 9 S102 to S109 are only illustrative examples. The process of establishing a 2.4 GHz channel between a mobile phone and a laptop computer may include: Fig. 9 For more frame interaction processes, the information, format and function carried by each frame can be referred to the description in the existing standards, and this application will not go into details.
[0273] S110, data transfer between mobile phone and laptop.
[0274] For example, the mobile phone can send data to the laptop computer through the established 2.4 GHz channel, so that the corresponding picture is displayed on the display window of the laptop computer.
[0275] For example, refer to Fig.11 , Fig.11 The figure is a schematic diagram of data transmission between a mobile phone and a laptop computer. For example, the mobile phone can send content displayed on the mobile phone interface to the laptop computer in the form of image frames via a 2.4 GHz channel. That is, the mobile phone can send image frames to the laptop computer via a 2.4 GHz channel, and the image frames can reflect the content displayed on the mobile phone interface.
[0276] For example, in Fig.11 In the example, the desktop displayed in the current interface 201 of the mobile phone is used for explanation. The desktop may include multiple status icons and multiple application icons. For example, the status icons may include: a mobile network signal icon, a Wi-Fi signal icon, a mobile phone battery icon, and a time icon. The application icons may include: a clock application icon, a calendar application icon, and the like.
[0277] Understandably, Fig.11 The content included on the mobile phone interface 201 is only exemplary. According to actual conditions, the mobile phone interface 201 may also include other content that may need to be used, and this application does not limit this.
[0278] The mobile phone generates an image frame 202 based on the interface 201. The mobile phone sends the image frame 202 to the laptop based on the connection between the mobile phone and the laptop. The image frame 202 may reflect the content displayed on the interface 201 of the mobile phone, and the laptop receives the image frame 202 sent by the mobile phone.
[0279] After the laptop computer is connected to the mobile phone, a screen projection interface 205 is displayed in the laptop computer interface 203. Exemplarily, the interface 203 also includes but is not limited to: an icon 204 of a screen projection recording application.
[0280] Among them, the screen projection recording application is an application installed in the laptop computer, which is used to record the video of the mobile phone projecting the screen to the laptop computer (the video of multiple image frames 202 sent by the mobile phone to the laptop computer) and to mark the recorded video with relevant information about the control operations (the control operations performed by the user on the mobile phone interface 201 and / or the user's control operations on the physical buttons on the mobile phone, etc.).
[0281] After receiving the image frame 202 sent by the mobile phone, the laptop computer can display the image frame 202 in the screen projection interface 205 of the interface 203 of the laptop computer.
[0282] In the embodiment of the present application, during the screen projection process, the user can operate on the mobile phone, such as inputting control operations, such as clicking, sliding, double-clicking, etc. The mobile phone responds to the control operation input by the user and displays the corresponding interface. In addition, the mobile phone periodically updates the screen based on the current interface (the interface may be different from the current interface). Fig.11 The interface in the laptop is different) to generate a corresponding image frame and send it to the laptop. Every time the laptop receives a new image frame sent by the mobile phone, the new image frame is displayed in the projection interface 205 to achieve collaborative operation between the laptop and the mobile phone.
[0283] In an embodiment of the present application, during the process of projecting the screen of a mobile phone to a laptop computer, at least one purpose of the user controlling the mobile phone is to test whether the functions of the mobile phone are normal.
[0284] For example, the user clicks on an application icon to test whether the mobile phone will start the application and display the content of the application (e.g., display the interface of the application) on the mobile phone interface 201. If the user clicks on the application icon and the mobile phone does not start the application or does not display the content of the application (e.g., the interface of the application) on the mobile phone interface 201, it means that the function of the mobile phone is abnormal.
[0285] For another example, the user long presses the icon of an application to test whether the mobile phone will display at least one control control (such as an uninstall control, a scan control, or a payment control, etc.) for controlling the application on the mobile phone interface 201. If the user long presses the icon of the application and the mobile phone does not display at least one control control for controlling the application on the mobile phone interface 201, it means that the function of the mobile phone is abnormal.
[0286] In the process of testing whether the functions of the mobile phone are normal, if the functions of the mobile phone are found to be abnormal, the abnormal situation of the mobile phone can be reproduced, so as to analyze the cause of the abnormal function of the mobile phone according to the reproduced situation, and then eliminate the abnormality according to the cause.
[0287] The user can use the screen projection recording application installed in the laptop to record the video of the mobile phone projecting the screen to the laptop, so as to reproduce the situation displayed on the interface 201 of the mobile phone in a period of time before and after the abnormal function of the mobile phone.
[0288] For example, the user can operate the laptop computer. The user can input a movement operation on the laptop computer, and the movement operation is used to move the cursor 206 displayed on the interface 203 of the laptop computer to the position where the icon 204 of the screen projection recording application displayed on the interface 203 of the laptop computer is located. Fig.12 , Fig.12 The schematic diagram of the interface of the laptop computer is shown as an example. In response to a moving operation, the laptop computer can move a cursor 206 displayed on the interface 203 of the laptop computer to the position of the icon 204 of the screen projection recording application displayed on the interface 203 of the laptop computer.
[0289] Thereafter, the user may input a manipulation operation on the icon 204 of the screen projection recording application through the cursor 206 , and the laptop computer may respond to the manipulation operation on the icon 204 of the screen projection recording application and then start the screen projection recording application.
[0290] For example, see Fig.13 , Fig.13 FIG. 2 is a schematic diagram of an interface of a laptop computer, in which an interface 207 of a screen projection recording application can be displayed on an interface 203 of the laptop computer. Fig.14 This is a schematic diagram of an interface of a screen projection recording application displayed in an interface of a laptop computer, see Fig.13 or Fig.14 ( Fig.14 for Fig.13), the interface 207 of the screen projection recording application displayed on the interface 203 of the laptop computer may include but is not limited to three areas, namely, a zoom-in / out closing area 2071, a recording setting area 2072, and a screen projection area 2073.
[0291] The shrink-enlarge-close area 2071 may include at least three controls, for example, a shrink control, an enlarge control, and a close control.
[0292] The zoom-out control is used to minimize the screen projection recording application interface 207. For example, if a user inputs a manipulation operation on the zoom-out control on a laptop computer, the laptop computer can minimize the screen projection recording application interface 207 in response to the manipulation operation on the zoom-out control.
[0293] The enlarged control is used to maximize the display of the screen projection recording application interface 207. For example, if the user inputs a manipulation operation on the enlarged control on the laptop, the laptop can maximize the display of the screen projection recording application interface 207 in response to the manipulation operation on the enlarged control.
[0294] The close control is used to close the screen projection recording application interface 207. For example, if the user inputs a manipulation operation on the close control on the laptop, the laptop can close the screen projection recording application interface 207 in response to the manipulation operation on the close control.
[0295] The recording setting area 2072 may include: a recording duration setting control, a manipulation playback duration setting control, a manipulation gesture setting control, a marking switch control, and a recording switch control.
[0296] The user can use the recording duration setting control to set: the duration of the recorded video output by the screen recording application, Fig.13 as well as Fig.14 The duration set by the recording duration setting control shown in the figure is 3 minutes.
[0297] The user can set the playback duration setting control to extend the duration of the video recorded by the screen projection recording application for scenes where the user performs input operations on the phone. Fig.13 as well as Fig.14 The duration set by manipulating the playback duration setting control shown in FIG. 1 is 3 seconds.
[0298] Through the control gesture setting control, users can set: the control gestures drawn in the video recorded by the screen projection recording application, the control gestures operated by the user on the mobile phone, etc.
[0299] Fig.13 and Fig.14In the example shown, the control gesture setting control may include 5 optional styles. The control gesture of the 4th finger from the left is framed, indicating that the control gesture of the 4th finger from the left is set. In this way, the control gesture drawn in the video recorded by the screen projection recording application and the control operation input by the user on the mobile phone is the control gesture of the 4th finger from the left.
[0300] The user can use the annotation switch control to set whether to annotate the control gestures entered by the user on the mobile phone and the description information of the control operations in the video recorded by the screen projection recording application. Fig.13 as well as Fig.14 The annotation switch control shown in is in the on state, indicating that the control gestures entered by the user on the mobile phone and the description information of the control operations need to be annotated in the video recorded by the screen projection recording application.
[0301] The user can use the recording switch control to set whether the screen projection recording application records the screen projection video. Fig.13 as well as Fig.14 The recording switch control shown in the figure is in the off state, indicating that the cast video is not currently being recorded.
[0302] The screen projection area 2073 is used to display the image frame 202 sent by the mobile phone.
[0303] After the laptop starts the screen projection recording application, the screen projection recording application can call the screen projection interface to obtain the image frame 202 sent by the mobile phone to the laptop during the screen projection process, and can display the image frame 202 sent by the mobile phone in the screen projection area 2073.
[0304] in, Fig.13 as well as Fig.14 The recording switch control shown in is in the off state, so the screen projection recording application does not record the projected video. If the user needs the screen projection recording application to start recording the projected video, the user can control the laptop computer. For example, the user can input a moving operation on the laptop computer, and the moving operation is used to move the cursor 206 displayed on the interface 203 of the laptop computer to: the position of the recording switch control in the recording setting area 2072 in the interface 207 of the screen projection recording application on the interface 203 of the laptop computer, see Fig.15 , Fig.15 The schematic diagram of the interface of the screen projection recording application displayed in the interface of the laptop computer is shown as an example. The laptop computer can move the cursor 206 displayed on the interface 203 of the laptop computer in response to the moving operation to: the position of the recording switch control in the recording setting area 2072 in the interface 207 of the screen projection recording application on the interface 203 of the laptop computer.
[0305] Afterwards, the user can input the operation of the recording switch control through the cursor 206. In response to the operation of the recording switch control, the laptop computer starts the function of recording the projected screen video of the screen projection recording application and starts video recording of the image frame 202 sent by the mobile phone displayed in the screen projection area 2073.
[0306] Also, see Fig.16 , Fig.16 The schematic diagram of the interface of the screen projection recording application displayed in the interface of the laptop computer is shown as an example. The laptop computer can also set the state of the recording switch control in the recording setting area 2072 to the on state so that the user knows that the screen projection recording application has started recording the projected video.
[0307] Afterwards, while the user is operating the mobile phone, the image frame displayed in the projection area 2073 in the interface 207 of the screen projection recording application changes with the changes in the content displayed on the interface 201 of the mobile phone.
[0308] Furthermore, when the user is operating the mobile phone, if the user finds that the mobile phone functions abnormally, the abnormal situation of the mobile phone can be reproduced through the recorded video, so as to analyze the cause of the abnormal situation of the mobile phone according to the reproduced situation, and then eliminate the abnormality according to the cause.
[0309] In one example, when a user manipulates a mobile phone, the mobile phone displays a corresponding interface in response to the manipulation operation input by the user. Fig.17 , Fig.17 This is a schematic diagram of an interface of a laptop computer, in which the mobile phone displays a gallery application interface in response to a manipulation operation input by a user in interface 201. The gallery application interface includes content corresponding to the gallery application.
[0310] Fig.17 The content included on the mobile phone interface 201 is only exemplary. According to actual conditions, the mobile phone interface 201 may also include other content that may need to be used, and this application does not limit this.
[0311] For example, Fig.18 This is a schematic diagram of an interface of a screen projection recording application displayed in an interface of a laptop computer, that is, Fig.18 for Fig.17 A partially enlarged image of the interface 207 of the screen projection recording application displayed on the interface 203 of the laptop computer. At this time, the projection area 2073 in the interface 207 of the screen projection recording application displays the content of the gallery application. Correspondingly, the content displayed on the interface 201 of the mobile phone is also the content of the gallery application.
[0312] Suppose that a user inputs a single-click operation on a photo icon displayed on the interface 201 of the mobile phone. Normally, the mobile phone will respond to the single-click operation and display a large picture of the photo on the interface 201 of the mobile phone. However, suppose that the mobile phone does not display a large picture of the photo on the interface 201 of the mobile phone in response to the single-click operation, that is, the mobile phone does not respond to the single-click operation, and the content displayed on the interface 201 of the mobile phone should have changed but actually did not change, which often indicates that the mobile phone is malfunctioning. The user can stop operating the mobile phone and use the screen projection recording application installed in the laptop to capture the video of the period before and after the malfunction of the mobile phone in the recorded video to reproduce the malfunction of the mobile phone.
[0313] in, Fig.17 as well as Fig.18 The recording switch control shown in is in the on state, so the screen projection recording application is recording the projected video. If the user needs to use the screen projection recording application installed in the laptop computer to capture the video within a period of time before and after the mobile phone's function is abnormal in the recorded video, the user can control the laptop computer. For example, the user can input a move operation on the laptop computer, and the move operation is used to move the cursor 206 displayed on the interface 203 of the laptop computer to: the position of the recording switch control in the recording setting area 2072 in the interface 207 of the screen projection recording application on the interface 203 of the laptop computer.
[0314] See also Fig.19 , Fig.19 The schematic diagram of the interface of the screen projection recording application displayed in the interface of the laptop computer is shown as an example. The laptop computer can move the cursor 206 displayed on the interface 203 of the laptop computer in response to the moving operation to: the position of the recording switch control in the recording setting area 2072 in the interface 207 of the screen projection recording application on the interface 203 of the laptop computer.
[0315] Afterwards, the user can input the operation of the recording switch control through the cursor 206. In response to the operation of the recording switch control, the laptop computer uses the screen projection recording application installed in the laptop computer to capture the video within a period of time before and after the abnormal function of the mobile phone in the recorded video.
[0316] Secondly, in another embodiment, video recording of the image frame 202 displayed in the projection area 2073 may no longer be performed.
[0317] Also, see Fig. 20 , Fig. 20The schematic diagram of the interface of the screen projection recording application displayed in the interface of the laptop computer is shown as an example. The laptop computer can set the state of the recording switch control to the off state so that the user can know that the screen projection recording application is no longer recording the video of the image frame 202 displayed in the projection area 2073.
[0318] Afterwards, when the user operates the mobile phone, the image frame displayed in the projection area 2073 changes as the content displayed on the interface 201 of the mobile phone changes.
[0319] Exemplarily, after the recording is finished, a prompt box may be displayed in the interface 207 to prompt that the recording has been finished. In addition, another prompt box may pop up in the interface 207 to display the save path of the video file corresponding to the video recording so that the operator can view the recorded video.
[0320] Among them, for "using the screen projection recording application installed in the laptop to capture the video within a period of time before and after the phone's function is malfunctioning", in the aforementioned illustrated example, in the recording duration setting control in the recording setting area 2072 in the interface 207 of the screen projection recording application, the set duration is 3 minutes, which is used to indicate that the duration of the recorded video output by the screen projection recording application can be 3 minutes.
[0321] In one embodiment, if the duration of the video recorded by the screen projection recording application is less than 3 minutes, the video recorded by the screen projection recording application with a duration of less than 3 minutes can be determined as a video within a period of time before and after the malfunction of the mobile phone.
[0322] Alternatively, in another embodiment, if the duration of the video recorded by the screen projection recording application is equal to 3 minutes, the video recorded by the screen projection recording application with a duration of 3 minutes can be determined as a video within a period of time before and after the mobile phone's function is abnormal.
[0323] Alternatively, in another embodiment, if the duration of the video recorded by the screen projection recording application is greater than 3 minutes, a video with a duration equal to 3 minutes can be captured from the video recorded by the screen projection recording application for a duration greater than 3 minutes and determined as the video within a period of time before and after the malfunction of the mobile phone.
[0324] In one example, the last 3 minutes of a video recorded by a screen projection recording application that is longer than 3 minutes can be captured.
[0325] For example, in a video recorded by the screen projection recording application with a duration longer than 3 minutes, the last image frame is used as the end image frame, and the image frame 3 minutes away from the last image frame is used as the start image frame. Then, in the video recorded by the screen projection recording application with a duration longer than 3 minutes, the image frames between the start image frame and the end image frame are determined, and the start image frame, the image frames between the start image frame and the end image frame, and the end image frame are combined in chronological order to obtain a 3-minute video before and after the mobile phone malfunctions.
[0326] For example, see Fig.21 , Fig.21 As an example, the distribution diagram of image frames in the recorded video is shown. The video recorded by the screen projection recording application may include multiple image frames, for example, it may include the 1st image frame, the 2nd image frame... the Mth image frame, the M+1th image frame... the Nth image frame, the N+1th image frame... the Pth image frame, the P+1th image frame... and the last image frame.
[0327] Among them, in the aforementioned embodiment, Fig.17 In the case where the interface 201 of the mobile phone shown in FIG. 18 has displayed the content in the gallery application, the user inputs a manipulation operation to the recording switch control through the cursor 206, and the last image frame of the video recorded by the screen projection recording application with a duration longer than 3 minutes can be Fig.17 or Fig.18 The image frame 202 is shown. Fig.17 or Fig.18 The content in the image frame 202 shown includes the content in the gallery application that has been displayed on the interface 201 of the mobile phone.
[0328] For example, see Fig. 22 , Fig. 22 Schematic diagram of capturing a video segment in a recorded video; the N+1th image frame ... the Pth image frame, the P+1th image frame ... and the last image frame are Fig.21 The last 3 minutes of image frames in the video shown, which is longer than 3 minutes, can be Fig.21 Among the multiple image frames shown, the N+1th image frame, the Pth image frame, the P+1th image frame, and the last image frame are intercepted, which are the image frames of the last 3 minutes, as the video of the period before and after the abnormal function of the mobile phone (see Fig.23 , Fig.23 , which is a distribution diagram of image frames in a segment of video captured for example), that is, the Nth image frame and image frames earlier than the Nth image frame are no longer needed.
[0329] Among them, it should be noted that, generally, during the process of user operating the mobile phone, if the user finds that the function of the mobile phone is abnormal, the user often stops operating the mobile phone, that is, no longer inputs the control operation in the mobile phone, and inputs the control operation of the recording switch control through the cursor 206 on the laptop computer, so that the laptop computer responds to the control operation of the recording switch control and captures the video of a period of time before and after the abnormal function of the mobile phone. In this way, the last 3 minutes of video captured in the recorded video with a recording duration greater than 3 minutes can often include the video of a period of time before and after the abnormal function of the mobile phone, which can reflect the situation displayed on the interface 201 of the mobile phone when the mobile phone is abnormal.
[0330] Among them, the captured last 3 minutes of video can reflect the content displayed on the interface 201 of the mobile phone in the last 3 minutes and the changes in the displayed content. However, in the last 3 minutes, the user often discovers the abnormal function of the mobile phone after operating the mobile phone, that is, in the last 3 minutes, the user actually inputs the control operation in the mobile phone, and the control operation is input in the mobile phone at a certain moment, and the input control operation triggers the abnormal function of the mobile phone. In this way, in another embodiment of the present application, after obtaining the last 3 minutes of video, the control position of the control operation actually input by the user in the mobile phone and the description information of the control operation can be perceived in the last 3 minutes of video. This can facilitate the subsequent analysis of what triggered the abnormal function of the mobile phone (triggered by which control operation input at which position on which interface), so as to analyze the cause of the abnormal function of the mobile phone.
[0331] In one embodiment of the present application, the control operations input by the user in the mobile phone may include a single-click operation, a double-click operation, a sliding operation, a long press operation, a two-finger zoom-in operation, and a two-finger zoom-out operation, etc.
[0332] For example, in one example, the user inputs a single-click operation on the icon 208 of the photo displayed on the interface 201 of the mobile phone at a certain moment, and the moment corresponds to the Pth image frame in the last 3 minutes of the video. In this way, a manipulation gesture can be drawn at the position where the icon 208 is located in the Pth image frame, for example, see Fig.24 , Fig.24 FIG. 1 is a schematic diagram for exemplarily illustrating annotating image frames in a captured video; Fig.13 and Fig.14 In the example shown, the manipulation gesture of the fourth finger from the left in the manipulation gesture setting control is framed, and the manipulation gesture of the fourth finger from the left in the manipulation gesture setting control can be drawn at the position where the icon 208 is located in the Pth image frame, and the description information of the manipulation operation is marked next to the manipulation gesture to obtain the P'th image frame, thereby obtaining Fig.25 The 3-minute video shown, Fig.25 The figure is a schematic diagram showing an exemplary embodiment of labeling image frames in a captured video.
[0333] The description information of the manipulation operation may include the manipulation type of the manipulation operation, or may also include the manipulation object targeted by the manipulation operation, or may also include the manipulation direction of the manipulation operation, or may also include the input time when the manipulation operation is input.
[0334] The manipulation types of the manipulation operation may include a single-click operation, a double-click operation, a slide operation, a long-press operation, a two-finger zoom-in operation, and a two-finger zoom-out operation, etc.
[0335] Afterwards, play Fig.25 In the 3-minute video shown, when playing to the P' image frame, the user can see the finger-style control gesture at the location of the icon 208 in the P' image frame, and can see the description information of the control operation displayed next to the control gesture, which is convenient for the user to analyze the reasons for the abnormal function of the mobile phone.
[0336] In another embodiment of the present application, the image frame (eg, the P'th image frame mentioned above) and the next image frame (eg, Figure 21 to Figure 25 In the example of FIG. 1 , a plurality of image frames are inserted between the image frames 100 and 101 , wherein the position of the icon 208 in the plurality of image frames is drawn with a finger-like manipulation gesture and annotated with description information of the manipulation operation. The next image frame is located after this image frame and adjacent to this image frame.
[0337] Through this embodiment, the control gestures of the control operations are respectively drawn at the control positions of the control operations in multiple image frames, and the descriptive information of the control operations are respectively marked. In this way, when the switching frequency of image frames is fast during video playing, the user's naked eyes can also perceive that the control positions of the control operations in multiple image frames respectively have the control gestures of the control operations and the descriptive information of the control operations, and then the descriptive information of the control operations can be used to analyze what triggers the abnormal function of the mobile phone (triggered by which control operation input at which position on which interface).
[0338] The duration of the inserted multiple image frames can be determined according to the duration set in the control playback duration setting control, for example, Fig.13 and Fig.14 In the example shown, the duration set in the playback duration setting control is 3 seconds, so the duration of the inserted multiple image frames can be 3 seconds.
[0339] For example, see Fig.26 , Fig.26The schematic diagram of inserting an image frame in a captured video is shown as an example; between the P'th image frame and the P+1th image frame, multiple image frames are inserted, and the position of the icon 208 in the multiple image frames is drawn with a finger-style control gesture and the description information of the control operation is marked next to the control gesture. The multiple image frames may include the P'1th image frame, the P'2th image frame, the P'3th image frame... and the P' X Image frames, etc.
[0340] Because in Fig.13 and Fig.14 In the example shown, the duration set in the playback duration setting control is 3 seconds, so the P'1 image frame, the P'2 image frame, the P'3 image frame, and the P'4 image frame are displayed. X The duration of the image frame can be 3 seconds, so as to obtain Fig. 27 The 3-minute + 3-minute video shown, Fig. 27 The figure is a distribution diagram of image frames in a video with image frames inserted therein, for example.
[0341] In the above example, the user inputs a single-click operation on the photo icon 208 displayed on the interface 201 of the mobile phone at a certain moment, and this moment corresponds to the Pth image frame in the last 3 minutes of the video.
[0342] And when the user inputs a single-click operation on the icon 208 of the photo displayed on the interface 201 of the mobile phone, the position of the single-click operation remains unchanged and is always located in the area where the icon 208 of the photo displayed on the interface 201 of the mobile phone is located.
[0343] In this case, the P'1th image frame, the P'2th image frame, the P'3th image frame, ... and the P'4th image frame among the inserted plurality of image frames. X The image frames may be respectively the same as the P'th image frame, that is, a plurality of P'th image frames are inserted between the P'th image frame and the P+1th image frame, and the duration of the inserted plurality of P'th image frames may be 3 seconds.
[0344] For example, see Fig.28 , Fig.28 The schematic diagram of the labeled image frame is shown as an example. At this moment, the user inputs a single-click operation on the icon 208 of the photo displayed on the interface 201 of the mobile phone. The control gesture 209 of the single-click operation is located in the area where the icon 208 of the photo in the P'th image frame is located. The description information 210 of the single-click operation is located on the left side of the control gesture 209 of the single-click operation. The description information 210 of the single-click operation may include "click the photo icon" to reflect that the control type of this control operation is a single-click operation, and the control object of the single-click operation is the photo icon.
[0345] In the aforementioned example, the user starts inputting a sliding operation to the left on the right side of the photo icon 208 displayed on the interface 201 of the mobile phone at a certain moment, and this moment corresponds to the Pth image frame in the last 3 minutes of the video.
[0346] And during the process of inputting the sliding operation, the position of the sliding operation keeps moving to the left.
[0347] In this case, the P'1th image frame, the P'2th image frame, the P'3th image frame, ... and the P'4th image frame among the inserted plurality of image frames. X The image frames may be different from the P'th image frames, respectively.
[0348] The P'1 image frame, the P'2 image frame, the P'3 image frame, and the P' X The image frames may differ from the P'th image frame in that: the positions of the manipulation gestures of the sliding operation in the image frames are different, the positions of the description information of the sliding operation in the image frames are different, and the contents of the description information of the sliding operation are not all the same.
[0349] In the P'1th image frame, the P'2th image frame, the P'3th image frame... and the P' X In the image frames, the position of the control gesture of the sliding operation in the image frames that are earlier in the order is closer to the starting position of the sliding operation, and the position of the control gesture of the sliding operation in the image frames that are later in the order is farther away from the starting position of the sliding operation.
[0350] For example, Figure 29 to Figure 32 Take this as an example to illustrate: Figures 29 to 31 They are schematic diagrams of image frames respectively.
[0351] Fig.29 is a schematic diagram of an illustrative labeled image frame; for example, Fig.29 It can be a schematic diagram of the P'th image frame. In which, the user starts to input a sliding operation of sliding to the left on the right side of the icon 208 of the photo displayed on the interface 201 of the mobile phone at a certain moment, and this moment corresponds to the P'th image frame in the last 3 minutes of the video. At this time, the control gesture 209 of the sliding operation is located in the blank area on the right side of the photo icon 208, and the description information 210 of the sliding operation is located on the left side of the control gesture 209 of the sliding operation. The content in the description information 210 of the sliding operation can include "start sliding to the left" to reflect that the control type of this control operation is a sliding operation, and the direction of the sliding operation is to the left.
[0352] Fig.30 is a schematic diagram of an illustrative labeled image frame; for example, Fig.30It can be "the P'1th image frame, the P'2th image frame, the P'3th image frame... and the P' X The non-P' in the image frame X A schematic diagram of one of the image frames, for example, Fig.30 is a schematic diagram of the P'3 image frame, Fig.30 This is a schematic diagram of the sliding operation process. Fig.30 In the figure, the control gesture 209 of the sliding operation is located in the area where the photo icon 208 is located, and the description information 210 of the sliding operation is located on the left side of the control gesture 209 of the sliding operation. The content in the description information 210 of the sliding operation may include "sliding to the left" to reflect that the control type of this control operation is a sliding operation, and the direction of the sliding operation is to the left.
[0353] Fig.31 is a schematic diagram of an illustrative labeled image frame; for example, Fig.31 It can be "the P'1th image frame, the P'2th image frame, the P'3th image frame... and the P' X The non-P' in the image frame X A schematic diagram of one of the image frames, Fig.31 The order of the image frames in Fig.30 After the order of the image frames in . For example, Fig.31 For P' 10 Schematic diagram of an image frame, Fig.31 This is a schematic diagram of the sliding operation process. Fig.31 In the figure, the control gesture 209 of the sliding operation is located in the area where the photo icon 211 is located, and the description information 210 of the sliding operation is located on the left side of the control gesture 209 of the sliding operation. The content in the description information 210 of the sliding operation may include "in the process of sliding to the left" to reflect that the control type of this control operation is a sliding operation, and the direction of the sliding operation is to the left.
[0354] Fig.32 is a schematic diagram of an illustrative labeled image frame; for example, Fig.32 It can be "the P'1th image frame, the P'2th image frame, the P'3th image frame... and the P' X The P'th X Schematic diagram of the image frame, it can be seen that Fig.32 The order of the image frames in Fig.31 After the sequence of image frames in . Fig.31 A schematic diagram of the end of the sliding operation. Fig.32, the control gesture 209 of the sliding operation is located in the area where the photo icon 212 is located, and the description information 210 of the sliding operation is located on the right side of the control gesture 209 of the sliding operation. The description information 210 of the sliding operation may include "End sliding to the left" to reflect that the control type of this control operation is a sliding operation, and the direction of the sliding operation is to the left, and the sliding has ended.
[0355] Among them, P' X X in the image frame is the number of inserted image frames, which can be a positive integer.
[0356] Figure 29 to Figure 32 This is merely an illustrative example of a sliding operation.
[0357] exist Fig.29 The P'th image frame shown is Fig.30 There are also P'1th image frame and P'2th image frame between the P'3th image frame, which will not be listed one by one.
[0358] exist Fig.30 The P'3 image frame shown is Fig.31 P' 10 There are multiple image frames between the image frames, which will not be listed one by one.
[0359] exist Fig.31 P' 10 Image frame and Fig.32 P' X There are multiple image frames between the image frames, which will not be listed one by one.
[0360] Fig.33 A schematic diagram of a data processing method of the present application is shown. The method can be applied to Figure 1 to Figure 4 Any of the screen projection scenarios shown in the present application is applied to the method Figure 1 The screen projection scenario between the mobile phone and the laptop computer is taken as an example for illustration, but is not intended to limit the scope of protection of the present application.
[0361] Among them, the method includes:
[0362] S301, the mobile phone captures an image frame and obtains the capture time when the image frame is captured, the image frame includes content displayed on the interface of the mobile phone when the image frame is captured.
[0363] In the present application, after the mobile phone turns on screen projection, the mobile phone can take screenshots of the mobile phone interface at a frequency of 120Hz, that is, 120 image frames can be captured per second, and the image frames contain the content displayed on the mobile phone interface.
[0364] In one embodiment of the present application, a mobile phone has a first screen projection module, which can be used to capture image frames and obtain the capture time when the image frames are captured.
[0365] S302, the mobile phone sends the image frame to the laptop computer and sends the capture time when the image frame is captured to the laptop computer.
[0366] In the present application, each time an image frame is captured and the capture time of the image frame is obtained, the mobile phone will send the captured image frame and the capture time of the image frame to the laptop computer.
[0367] For example, the mobile phone can send the captured image frame and the obtained capture time when the image frame is captured to the laptop computer through the Wi-Fi connection established between the mobile phone and the laptop computer.
[0368] In one embodiment of the present application, a mobile phone has a first screen projection module and a laptop computer has a second screen projection module. The first screen projection module in the mobile phone can send image frames and the capture time when the image frame is captured to the second screen projection module in the laptop computer.
[0369] S303, the notebook computer receives the image frame and the capture time when the image frame is captured.
[0370] In one embodiment of the present application, a laptop computer has a second screen projection module, which can be used to receive the image frame sent by the first screen projection module in the mobile phone and the capture time when the image frame is captured.
[0371] S304, the notebook computer displays the image frame on the interface of the notebook computer, and stores the corresponding relationship between the identifier of the image frame and the capture time when the image frame is captured.
[0372] In one embodiment of the present application, the second screen projection module in the laptop computer may display the image frame on the interface of the laptop computer.
[0373] In one embodiment of the present application, the laptop computer also has a first storage module, which can be a second screen projection module in the laptop computer that sends the identification of the image frame and the capture time when the image frame is captured to the first storage module, and then the first storage module can store the correspondence between the identification of the image frame and the capture time when the image frame is captured.
[0374] Different image frames have different identifiers.
[0375] S305, the laptop computer receives a start operation input by the user, where the start operation is used to start the screen projection recording application.
[0376] During the process of projecting the screen of a mobile phone to a laptop computer, at least one purpose of the user controlling the mobile phone is to test whether the functions of the mobile phone are normal.
[0377] In the process of testing whether the functions of the mobile phone are normal, if the functions of the mobile phone are found to be abnormal, the abnormal situation of the mobile phone can be reproduced, so as to analyze the cause of the abnormal function of the mobile phone according to the reproduced situation, and then eliminate the abnormality according to the cause.
[0378] For example, a user can use a screen projection recording application installed in a laptop to record a video of a mobile phone projecting the screen to the laptop to reproduce the situation displayed on the mobile phone's interface for a period of time before and after the mobile phone's malfunction.
[0379] For example, a user may input a startup operation on a laptop computer, and the startup operation is used to start a screen projection recording application. The laptop computer may receive the startup operation input by the user.
[0380] In one embodiment of the present application, the notebook computer has a UI (User Interface) module, and the UI module in the notebook computer may receive a startup operation input by a user.
[0381] S306, the laptop computer starts the screen projection recording application in response to the start-up operation, and displays the interface of the screen projection recording application on the interface of the laptop computer.
[0382] In one embodiment of the present application, the UI module in the laptop computer may start the screen projection recording application in response to the startup operation, and display the interface of the screen projection recording application on the interface of the laptop computer.
[0383] S307, the laptop computer displays the screen recording setting controls and the image frame on the interface of the screen projection recording application.
[0384] In one embodiment of the present application, the UI module in the laptop computer may display the screen recording setting controls and image frames on the interface of the screen projection recording application.
[0385] S308, the laptop receives a setting operation input by the user according to the screen recording setting control, where the setting operation is used to at least set the duration of the recorded video.
[0386] In the present application, the user can set at least the duration of the recorded video through the screen recording setting control. If the user needs to set at least the duration of the recorded video through the screen recording setting control, the user can input a setting operation according to the screen recording setting control on the laptop computer, and the setting operation is used to set at least the duration of the recorded video. The laptop computer can receive the setting operation input by the user according to the screen recording setting control.
[0387] In one embodiment of the present application, the UI module in the laptop computer may receive the setting operation input by the user according to the screen recording setting control.
[0388] S309: The notebook computer at least sets the duration of the recorded video in response to the setting operation.
[0389] In one embodiment of the present application, the screen projection recording application includes a recording module. The UI module in the laptop computer can respond to the setting operation and send relevant information of the setting operation to the recording module in the screen projection recording application. The recording module in the screen projection recording application can set the duration of the recorded video according to the relevant information of the setting operation.
[0390] S310, the laptop computer receives a start screen recording operation input by the user according to the screen recording setting control, where the start screen recording operation is used to at least control the screen projection recording application to start recording the projected screen video.
[0391] In the present application, after the screen projection recording application is started, the projected video has not been recorded. If the user needs the screen projection recording application to start recording the projected video, the user can control the laptop computer. For example, the user can input the start screen recording operation on the laptop computer according to the screen recording setting control, and the laptop computer can receive the start screen recording operation input by the user according to the screen recording setting control.
[0392] In one embodiment of the present application, the UI module in the laptop computer may receive a screen recording start operation input by the user according to a screen recording setting control.
[0393] S311, in response to the start of the screen recording operation, the laptop computer starts recording the projected video, and the recorded video includes image frames displayed on the interface of the screen projection recording application.
[0394] In one embodiment of the present application, the UI module in the laptop computer can send relevant information for starting the screen recording operation to the recording module in the screen projection recording application, and the recording module in the screen projection recording application can start recording the projected video based on the relevant information for starting the screen recording operation, for example, recording a video of the image frame displayed in the interface of the screen projection recording application.
[0395] S312, the mobile phone obtains the control operation input by the user in the mobile phone.
[0396] Furthermore, when the user is operating the mobile phone, if the user finds that the mobile phone functions abnormally, the abnormal situation of the mobile phone can be reproduced through the recorded video, so as to analyze the cause of the abnormal situation of the mobile phone according to the reproduced situation, and then eliminate the abnormality according to the cause.
[0397] When a user is operating a mobile phone, the mobile phone can obtain the control operations input by the user in the mobile phone. The control operations include the control operations on the physical buttons in the mobile phone, and can also include interactive operations input on the UI interface.
[0398] In one embodiment of the present application, the mobile phone has an event module, and the event module in the mobile phone can obtain the control operations input by the user in the mobile phone.
[0399] S313, the mobile phone responds to the manipulation operation to perform a corresponding action, and the mobile phone obtains manipulation parameters of the manipulation operation, the manipulation parameters at least including the input time when the manipulation operation is input, the manipulation position of the manipulation operation, and the manipulation type of the manipulation operation.
[0400] Alternatively, the manipulation parameter may further include a manipulation object targeted by the manipulation operation, or may further include a manipulation direction of the manipulation operation, etc.
[0401] In one embodiment of the present application, the event module in the mobile phone may respond to the manipulation operation and perform corresponding actions for the manipulation operation. Secondly, the event module in the mobile phone may obtain the manipulation parameters of the manipulation operation.
[0402] The manipulation types of the manipulation operation may include a single-click operation, a double-click operation, a slide operation, a long-press operation, a two-finger zoom-in operation, and a two-finger zoom-out operation, etc.
[0403] The control position of the control operation includes the position coordinates of the control operation in the interface of the mobile phone or the position of the physical button targeted by the control operation.
[0404] S314, the mobile phone sends control parameters to the laptop computer.
[0405] The screen projection and recording application in the laptop computer has a second storage module.
[0406] Thus, in one embodiment of the present application, the event module in the mobile phone may send control parameters to the second storage module in the screen projection and recording application in the laptop computer.
[0407] S315, the laptop computer receives the control parameters.
[0408] In one embodiment of the present application, the second storage module in the screen projection and recording application in the laptop computer may receive the control parameters sent by the event module of the mobile phone.
[0409] S316, the notebook computer stores the control parameters.
[0410] In one embodiment of the present application, the second storage module in the screen projection and recording application in the laptop computer may store the control parameters sent by the event module of the mobile phone.
[0411] S317, the laptop computer receives an operation of ending screen recording input by the user according to the screen recording setting control.
[0412] In one example, while the user is operating the mobile phone, the content displayed in the mobile phone interface may change. In one example, suppose that the user inputs a single-click operation on a photo icon displayed on the mobile phone interface. Normally, the mobile phone will respond to the single-click operation and display a large picture of the photo in the mobile phone interface. However, suppose that the mobile phone does not display a large picture of the photo in the mobile phone interface in response to the single-click operation, that is, the mobile phone does not respond to the single-click operation, and the content displayed on the mobile phone interface should have changed but actually did not change, which often indicates that the mobile phone is malfunctioning. At this time, the user can use the screen projection recording application installed in the laptop to capture the video of the period before and after the malfunction of the mobile phone in the recorded video to reproduce the malfunction of the mobile phone.
[0413] Corresponding operations may be input into the laptop computer. For example, in one example, the corresponding operation may be an operation to end screen recording, etc., so as to capture in the recorded video a video of a period of time before and after the abnormal function of the mobile phone.
[0414] For example, the user may input an end screen recording operation according to the screen recording setting control on the laptop computer, and the laptop computer may receive the end screen recording operation input by the user according to the screen recording setting control.
[0415] In one embodiment of the present application, the UI module in the laptop computer may receive the end screen recording operation input by the user according to the screen recording setting control.
[0416] S318, the laptop computer responds to the end of the screen recording operation and ends recording the projected screen video.
[0417] In one embodiment of the present application, the UI module in the laptop computer can send relevant information about ending the screen recording operation to the recording module in the screen projection recording application, and the recording module in the screen projection recording application can end recording the projected screen video based on the relevant information about ending the screen recording operation, that is, video recording of the image frames displayed on the interface of the screen projection recording application can no longer continue.
[0418] Afterwards, while the user is operating the mobile phone, the image frame displayed on the interface of the screen projection recording application changes as the content displayed on the interface of the mobile phone changes.
[0419] S319, the laptop computer captures a video clip at the end of the recorded video and having a preset duration.
[0420] In one embodiment of the present application, a recording module in a screen projection recording application in a laptop computer may capture a video clip at the end of a recorded video with a preset duration.
[0421] It should be noted that, generally, when a user is operating a mobile phone and finds that the mobile phone is malfunctioning, the user often stops operating the mobile phone, that is, stops inputting control operations in the mobile phone, and controls the laptop computer to capture a video clip at the end of the recorded video with a preset duration. In this way, the captured video clip includes the video of a period of time before and after the mobile phone malfunctions, that is, the captured video can reflect the display of the mobile phone interface a period of time before and after the mobile phone malfunctions.
[0422] The preset duration can be determined according to the settings set by the user in the screen recording settings control, for example, the preset duration is 3 minutes.
[0423] S320, the notebook computer obtains the identifier of the image frame in the corresponding relationship between the identifier of the image frame and the capture time when the image frame is captured according to the input time in the stored control parameters.
[0424] In one embodiment of the present application, the recording module in the screen projection recording application in the laptop computer may send a control parameter acquisition request to the second storage module in the screen projection recording application in the laptop computer, so that the second storage module sends the control parameters stored in S316 to the recording module according to the control parameter acquisition request, and the recording module receives the control parameters stored in S316 sent by the second storage module according to the control parameter acquisition request.
[0425] The recording module then passes the input time in the control parameters to the first storage module in the laptop computer, so that the first storage module uses the input time in the control parameters as the interception time, searches for the correspondence between the identifier of the image frame stored in S304 and the interception time when the image frame is intercepted, and then obtains the identifier of the image frame in the correspondence between the identifier of the image frame stored in S304 and the interception time when the image frame is intercepted, and sends the obtained identifier of the image frame to the recording module. The recording module can receive the identifier of the image frame sent by the first storage module.
[0426] S321, the laptop computer indexes the image frame corresponding to the identifier in the video clip, and marks the stored control parameters in the image frame corresponding to the identifier to obtain the target video.
[0427] The recording module in the screen projection recording application in the laptop computer obtains the control parameters stored in S316 in S320, and the control parameters at least include the input time when the control operation is input, the control position of the control operation, and the control type of the control operation.
[0428] For example, at the manipulation position of the manipulation operation in the image frame corresponding to the identifier, the manipulation gesture of the manipulation operation is drawn, and the description information of the manipulation operation is marked next to the manipulation gesture of the manipulation operation, so as to obtain the target video.
[0429] The description information of the manipulation operation may include the manipulation type of the manipulation operation, or may also include the manipulation object targeted by the manipulation operation, or may also include the manipulation direction of the manipulation operation, or may also include the input time when the manipulation operation is input.
[0430] Afterwards, during the process of playing the target video, when the image frame corresponding to the logo is played, the user can see the control gesture at the control position of the control operation in the image frame corresponding to the logo, and can see the description information of the control operation displayed next to the control gesture, which is convenient for the user to analyze the reasons for the abnormal functions of the mobile phone.
[0431] In one embodiment of the present application, a recording module in a screen projection recording application in a laptop computer may index and search for an image frame corresponding to the identifier in a captured video clip, mark the stored control parameters in the image frame corresponding to the identifier, and obtain a target video.
[0432] In another embodiment of the present application, multiple image frames can be inserted between this image frame (for example, the image frame corresponding to the aforementioned identifier) and the next image frame, and the control positions in the inserted multiple image frames are drawn with control gestures of the control operations and marked with descriptive information of the control operations. The next image frame is located after this image frame and adjacent to this image frame.
[0433] Through another embodiment of the present application, control gestures of control operations are drawn and descriptive information of the control operations are marked at control positions in multiple image frames. In this way, when the switching frequency of image frames is fast during video playing, the user's naked eyes can also perceive that the control positions in multiple image frames respectively have control gestures of control operations and descriptive information of control operations, and then can analyze what triggers the abnormal function of the mobile phone (triggered by which control operation input at which position on which interface).
[0434] The duration of the inserted multiple image frames can be determined according to the user's settings in the screen recording settings control. For example, the duration is 3 seconds. In this way, the duration of the inserted multiple image frames can be 3 seconds.
[0435] Fig.34 A schematic diagram of a data processing method of the present application is shown. The method can be applied to Figure 1 to Figure 4 Any of the screen projection scenarios shown in the present application is applied to the method Figure 1 The screen projection scenario between the mobile phone and the laptop computer is taken as an example for illustration, but is not intended to limit the scope of protection of the present application.
[0436] The method includes S401 to S413. For S401 to S413, reference may be made to the description of S301 to S313, which will not be described in detail here.
[0437] Furthermore, the method further comprises:
[0438] S414, the mobile phone sends control parameters to the laptop computer.
[0439] The screen projection and recording application in the laptop computer has a recording module.
[0440] Thus, in one embodiment of the present application, the event module in the mobile phone may send control parameters to the recording module in the screen projection recording application in the laptop computer.
[0441] S415, the laptop computer receives the control parameters.
[0442] In one embodiment of the present application, the recording module in the screen projection recording application in the laptop computer may receive the control parameters sent by the event module of the mobile phone.
[0443] S416, the laptop computer obtains the identifier of the image frame in the corresponding relationship between the identifier of the image frame and the capture time when the image frame is captured according to the input time in the control parameter.
[0444] In one embodiment of the present application, a recording module in a screen projection recording application in a laptop computer may pass the input time in the control parameters to a first storage module in the laptop computer, so that the first storage module uses the input time in the control parameters as the capture time, searches for the correspondence between the identifier of the stored image frame and the capture time when the image frame is captured, and then obtains the identifier of the image frame in the correspondence between the identifier of the stored image frame and the capture time when the image frame is captured, and sends the obtained identifier of the image frame to the recording module. The recording module can receive the identifier of the image frame sent by the first storage module.
[0445] S417, the laptop computer indexes the image frame corresponding to the identifier in the recorded video, and marks the control parameters in the image frame corresponding to the identifier.
[0446] The control parameters at least include an input time when a control operation is input, a control position of the control operation, and a control type of the control operation.
[0447] For example, at the manipulation position of the manipulation operation in the image frame corresponding to the identifier, a manipulation gesture of the manipulation operation is drawn, and description information of the manipulation operation is marked next to the manipulation gesture of the manipulation operation.
[0448] The description information of the manipulation operation may include the manipulation type of the manipulation operation, or may also include the manipulation object targeted by the manipulation operation, or may also include the manipulation direction of the manipulation operation, or may also include the input time when the manipulation operation is input.
[0449] Afterwards, when playing the recorded video, when the image frame corresponding to the logo is played, the user can see the control gesture at the control position of the control operation in the image frame corresponding to the logo, and can see the description information of the control operation displayed next to the control gesture, which is convenient for the user to analyze the reasons for the abnormal functions of the mobile phone.
[0450] In one embodiment of the present application, a recording module in a screen projection recording application in a laptop computer may index and search for an image frame corresponding to the identifier in the recorded video, and mark the control parameters in the image frame corresponding to the identifier.
[0451] In another embodiment of the present application, multiple image frames can be inserted between this image frame (for example, the image frame corresponding to the aforementioned identifier) and the next image frame, and the control positions in the inserted multiple image frames are drawn with control gestures of the control operations and marked with descriptive information of the control operations. The next image frame is located after this image frame and adjacent to this image frame.
[0452] Through another embodiment of the present application, control gestures of control operations are drawn and descriptive information of the control operations are marked at control positions in multiple image frames. In this way, when the switching frequency of image frames is fast during video playing, the user's naked eyes can also perceive that the control positions in multiple image frames respectively have control gestures of control operations and descriptive information of control operations, and then can analyze what triggers the abnormal function of the mobile phone (triggered by which control operation input at which position on which interface).
[0453] The duration of the inserted multiple image frames can be determined according to the user's settings in the screen recording settings control. For example, the duration is 3 seconds. In this way, the duration of the inserted multiple image frames can be 3 seconds.
[0454] S418, the laptop computer receives an operation of ending screen recording input by the user according to the screen recording setting control.
[0455] In one example, while the user is operating the mobile phone, the content displayed in the mobile phone interface may change. In one example, suppose that the user inputs a single-click operation on a photo icon displayed on the mobile phone interface. Normally, the mobile phone will respond to the single-click operation and display a large picture of the photo in the mobile phone interface. However, suppose that the mobile phone does not display a large picture of the photo in the mobile phone interface in response to the single-click operation, that is, the mobile phone does not respond to the single-click operation, and the content displayed on the mobile phone interface should have changed but actually did not change, which often indicates that the mobile phone is malfunctioning. At this time, the user can use the screen projection recording application installed in the laptop to capture the video of the period before and after the malfunction of the mobile phone in the recorded video to reproduce the malfunction of the mobile phone.
[0456] Corresponding operations may be input into the laptop computer. For example, in one example, the corresponding operation may be an operation to end screen recording, etc., so as to capture in the recorded video a video of a period of time before and after the abnormal function of the mobile phone.
[0457] For example, the user may input an end screen recording operation according to the screen recording setting control on the laptop computer, and the laptop computer may receive the end screen recording operation input by the user according to the screen recording setting control.
[0458] In one embodiment of the present application, the UI module in the laptop computer may receive the end screen recording operation input by the user according to the screen recording setting control.
[0459] S419, the laptop computer responds to the end of the screen recording operation and ends recording the projected screen video.
[0460] In one embodiment of the present application, the UI module in the laptop computer can send relevant information about ending the screen recording operation to the recording module in the screen projection recording application, and the recording module in the screen projection recording application can end recording the projected screen video based on the relevant information about ending the screen recording operation, that is, video recording of the image frames displayed on the interface of the screen projection recording application can no longer continue.
[0461] Afterwards, while the user is operating the mobile phone, the image frame displayed on the interface of the screen projection recording application changes as the content displayed on the interface of the mobile phone changes.
[0462] S420, the notebook computer captures a video clip at the end of the recorded video and having a preset duration to obtain a target video.
[0463] In one embodiment of the present application, a recording module in a screen projection recording application in a laptop computer may capture a video clip at the end of a recorded video with a preset duration.
[0464] It should be noted that, generally, when a user is operating a mobile phone and finds that the mobile phone is malfunctioning, the user often stops operating the mobile phone, that is, stops inputting control operations in the mobile phone, and controls the laptop computer to capture a video clip at the end of the recorded video with a preset duration to obtain a target video. The target video includes the video of a period of time before and after the mobile phone is malfunctioning, that is, the target video can reflect the display status of the mobile phone interface in a period of time before and after the mobile phone is malfunctioning.
[0465] The preset duration can be determined according to the settings set by the user in the screen recording settings control, for example, the preset duration is 3 minutes.
[0466] It is understandable that, in order to realize the above functions, the electronic device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of the present application.
[0467] In one example, Fig.35 A schematic block diagram of a device 200 according to an embodiment of the present application is shown. The device 200 may include: a processor 201 and a transceiver / transceiver pin 202 , and optionally, may further include a memory 203 .
[0468] The components of the device 200 are coupled together via a bus 204, wherein the bus 204 may include a power bus, a control bus, and a status signal bus in addition to a data bus. However, for the sake of clarity, various buses are referred to as bus 204 in the figure.
[0469] Optionally, the memory 203 may be used for the instructions in the aforementioned method embodiment. The processor 201 may be used to execute the instructions in the memory 203, and control the receiving pin to receive a signal, and control the sending pin to send a signal.
[0470] The apparatus 200 may be the electronic device or a chip of the electronic device in the above method embodiment.
[0471] Among them, all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here.
[0472] This embodiment also provides a computer storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the data processing method in the above-mentioned embodiment.
[0473] This embodiment also provides a computer program product. When the computer program product runs on a computer, it enables the computer to execute the above-mentioned related steps to implement the data processing method in the above-mentioned embodiment.
[0474] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer execution instructions, and when the device is running, the processor can execute the computer execution instructions stored in the memory so that the chip executes the data processing method in the above-mentioned method embodiments.
[0475] Among them, the electronic device, computer storage medium, computer program product or chip provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above and will not be repeated here.
[0476] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.
[0477] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0478] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0479] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0480] Any content of each embodiment of the present application, as well as any content of the same embodiment, can be freely combined. Any combination of the above content is within the scope of the present application.
[0481] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and may include several instructions to enable a device (which may be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium may include: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.
[0482] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.
[0483] The steps of the method or algorithm described in conjunction with the disclosure of the embodiments of the present application can be implemented in a hardware manner, or can be implemented by a processor executing a software instruction. The software instruction can be composed of corresponding software modules, and the software module can be stored in a random access memory (Random Access Memory, RAM), a flash memory, a read-only memory (Read Only Memory, ROM), an erasable programmable read-only memory (Erasable Programmable ROM, EPROM), an electrically erasable programmable read-only memory (Electrically EPROM, EEPROM), a register, a hard disk, a mobile hard disk, a read-only compact disk (CD-ROM) or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a network device. Of course, the processor and the storage medium can also be present in a network device as discrete components.
[0484] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented with hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media may include computer storage media and communication media, wherein the communication media may include any medium that is convenient for transmitting a computer program from one place to another. The storage medium may be any available medium that a general or special-purpose computer can access.
[0485] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.
Claims
1. A data processing method, characterized in that: Applied to a data processing system, the data processing system includes a first electronic device and a second electronic device, the first electronic device projects a screen to the second electronic device through a communication connection, and the second electronic device records a video of the first electronic device projecting the screen to the second electronic device; The method comprises: During the screen projection of the first electronic device to the second electronic device, the first electronic device obtains a manipulation parameter of the manipulation operation in response to a manipulation operation on the first electronic device; the manipulation parameter at least includes an input time when the manipulation operation is input, a manipulation position of the manipulation operation, and a manipulation type of the manipulation operation; The first electronic device sends the control parameter to the second electronic device; The second electronic device receives the control parameter; The second electronic device searches for an image frame corresponding to the input moment in the video recorded by the second electronic device; and at least marks a manipulation position and a manipulation type of the manipulation operation in the image frame corresponding to the input moment.
2. The method according to claim 1, characterized in that The method further comprises: The second electronic device captures a video segment including the marked image frame in the recorded video in response to the manipulation operation on the second electronic device.
3. The method according to claim 2, characterized in that The second electronic device intercepts a video segment including the marked image frame in the recorded video in response to the manipulation operation on the second electronic device, including: In response to a manipulation operation on the second electronic device, the second electronic device captures a video segment at the end of the recorded video with a preset duration, and the video segment at the end of the recorded video with a preset duration includes the marked image frame.
4. The method according to claim 1, characterized in that: The second electronic device marks at least a manipulation position of the manipulation operation and a manipulation type of the manipulation operation in the image frame corresponding to the input moment, including: The second electronic device draws a manipulation gesture at the manipulation position of the manipulation operation in the image frame corresponding to the input moment, and at least marks the manipulation type of the manipulation operation next to the manipulation gesture.
5. The method according to claim 4, characterized in that The second electronic device draws a manipulation gesture at the manipulation position of the manipulation operation in the image frame corresponding to the input moment, including: The second electronic device selects a manipulation gesture that matches the manipulation type of the manipulation operation from a plurality of preset manipulation gestures; The second electronic device draws the adapted manipulation gesture at the manipulation position.
6. The method according to claim 1, characterized in that The method further comprises: The second electronic device acquires a plurality of image frames to be inserted; the image frames to be inserted are marked with at least a manipulation position of the manipulation operation and a manipulation type of the manipulation operation; The second electronic device inserts the plurality of image frames to be inserted between the marked image frame and the next image frame in the recorded video; The next image frame is located after the marked image frame and is adjacent to the marked image frame.
7. The method according to claim 6, characterized in that The manipulation operation on the first electronic device includes a manipulation operation in which the manipulation position does not change; The second electronic device acquires a plurality of image frames to be inserted, including: The second electronic device generates the multiple image frames to be inserted according to the marked image frames.
8. The method according to claim 7, characterized in that The second electronic device generates a plurality of image frames to be inserted according to the marked image frames, including: The second electronic device makes a plurality of copies of the marked image frame; The second electronic device obtains the multiple image frames to be inserted according to the multiple copies of the marked image frames.
9. The method according to claim 6, characterized in that The manipulation operation on the first electronic device is a manipulation operation in which the manipulation position gradually changes; the manipulation parameters of the manipulation operation include multiple manipulation positions of the manipulation operation, and the multiple manipulation positions have a sequence; The second electronic device acquires a plurality of image frames to be inserted, including: The second electronic device generates a plurality of image frames to be annotated according to the image frame corresponding to the input moment, and the plurality of image frames to be annotated have a sequence; The second electronic device marks the corresponding manipulation position and the manipulation type of the manipulation operation in each image frame to be marked according to the sequence between the multiple manipulation positions and the sequence between the multiple image frames to be marked, so as to obtain the multiple image frames to be inserted; The earlier the sequence of the marked manipulation positions is, the earlier the sequence of the marked image frames to be marked is; and the later the sequence of the marked manipulation positions is, the later the sequence of the marked image frames to be marked is.
10. The method according to claim 9, characterized in that The second electronic device generates a plurality of image frames to be annotated according to the image frame corresponding to the input time, including: The second electronic device makes a plurality of copies of the image frame corresponding to the input time; The second electronic device obtains the multiple image frames to be marked based on the multiple copies of the image frames corresponding to the input moments.
11. The method according to claim 1, characterized in that: The method further comprises: The first electronic device sends the interception time of the image frame of the projected screen to the second electronic device, where the interception time includes the time when the first electronic device intercepts the image frame of the projected screen; The second electronic device receives the interception time; and stores the identifier of the image frame of the projected screen and the interception time in the correspondence between the identifier of the image frame and the interception time; The second electronic device searches for an image frame corresponding to the input time in the video recorded by the second electronic device, including: The second electronic device uses the input time as the interception time, and searches for the image frame corresponding to the input time in the corresponding relationship.
12. A data processing method, characterized in that: A first electronic device is applied to a data processing system, the data processing system further comprising a second electronic device, the first electronic device projects a screen to the second electronic device through a communication connection, the second electronic device records a video of the first electronic device projecting the screen to the second electronic device, the method comprising: In the process of projecting the screen of the first electronic device to the second electronic device, in response to a manipulation operation on the first electronic device, a manipulation parameter of the manipulation operation is acquired; the manipulation parameter at least includes an input time when the manipulation operation is input, a manipulation position of the manipulation operation, and a manipulation type of the manipulation operation; The control parameters are sent to the second electronic device so that the second electronic device searches for an image frame corresponding to the input moment in the video recorded by the second electronic device; in the image frame corresponding to the input moment, at least a control position of the control operation and a control type of the control operation are marked.
13. A data processing method, characterized in that: A second electronic device applied to a data processing system, the data processing system further comprising a first electronic device, the first electronic device casting a screen to the second electronic device via a communication connection, the second electronic device recording a video of the first electronic device casting a screen to the second electronic device, the method comprising: receiving a manipulation parameter of a manipulation operation sent by the first electronic device to the second electronic device; the manipulation parameter at least includes an input time when the manipulation operation is input, a manipulation position of the manipulation operation, and a manipulation type of the manipulation operation; the manipulation parameter is acquired by the first electronic device in response to the manipulation operation on the first electronic device during the process of projecting the screen of the first electronic device to the second electronic device; Searching for an image frame corresponding to the input moment in the video recorded by the second electronic device; In the image frame corresponding to the input moment, at least the manipulation position of the manipulation operation and the manipulation type of the manipulation operation are marked.
14. A first electronic device, characterized in that: include: One or more processors, memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and when the computer programs are executed by the one or more processors, the first electronic device performs the following steps: In a process in which a first electronic device projects a screen onto a second electronic device and the second electronic device records a video of the projected screen, in response to a manipulation operation on the first electronic device, a manipulation parameter of the manipulation operation is acquired; the manipulation parameter at least includes an input time when the manipulation operation is input, a manipulation position of the manipulation operation, and a manipulation type of the manipulation operation; Sending the control parameter to the second electronic device, so that the second electronic device searches for an image frame corresponding to the input moment in the video recorded by the second electronic device; In the image frame corresponding to the input moment, at least the manipulation position of the manipulation operation and the manipulation type of the manipulation operation are marked.
15. The first electronic device according to claim 14, characterized in that: When the program instructions are executed by the processor, the first electronic device executes the following steps: Sending the capture time of the image frame of the projected screen to the second electronic device, where the capture time includes the time when the first electronic device captures the image frame of the projected screen; So that the second electronic device receives the interception time; in the correspondence between the identifier of the image frame and the interception time, the identifier of the projected image frame and the interception time are stored; and when searching for the image frame corresponding to the input time in the video recorded by the second electronic device, the input time is used as the interception time, and the image frame corresponding to the input time is searched in the correspondence.
16. A second electronic device, characterized in that: include: One or more processors, memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and when the computer programs are executed by the one or more processors, the second electronic device performs the following steps: In a process in which a first electronic device projects a screen onto a second electronic device and the second electronic device records a video of the projected screen, receiving a control parameter of a control operation sent by the first electronic device to the second electronic device; the control parameter at least includes an input time when the control operation is input, a control position of the control operation, and a control type of the control operation; The manipulation parameter is acquired by the first electronic device in response to the manipulation operation on the first electronic device during the process of the first electronic device sending the projection image frame to the second electronic device; Searching for an image frame corresponding to the input moment in the video recorded by the second electronic device; In the image frame corresponding to the input moment, at least the manipulation position of the manipulation operation and the manipulation type of the manipulation operation are marked.
17. The second electronic device according to claim 16, characterized in that: When the program instructions are executed by the processor, the second electronic device executes the following steps: In response to a manipulation operation on the second electronic device, a video segment including the marked image frames is captured from the recorded video.
18. The second electronic device according to claim 17, characterized in that: When the program instructions are executed by the processor, the second electronic device executes the following steps: In response to the manipulation of the second electronic device, in the recorded video, a video segment at the end with a preset duration is captured, and the video segment at the end with a preset duration includes the marked image frame.
19. The second electronic device according to claim 16, characterized in that: When the program instructions are executed by the processor, the second electronic device executes the following steps: A manipulation gesture is drawn at the manipulation position of the manipulation operation in the image frame corresponding to the input moment, and at least the manipulation type of the manipulation operation is marked next to the manipulation gesture.
20. The second electronic device according to claim 19, characterized in that: When the program instructions are executed by the processor, the second electronic device executes the following steps: Selecting a manipulation gesture that matches the manipulation type of the manipulation operation from among a plurality of preset manipulation gestures; The adapted manipulation gesture is drawn at the manipulation position.
21. The second electronic device according to claim 16, characterized in that: When the program instructions are executed by the processor, the second electronic device executes the following steps: Acquire a plurality of image frames to be inserted; the image frames to be inserted are at least marked with a manipulation position of the manipulation operation and a manipulation type of the manipulation operation; Inserting the plurality of image frames to be inserted between the marked image frame and the next image frame in the recorded video; The next image frame is located after the marked image frame and is adjacent to the marked image frame.
22. The second electronic device according to claim 21, characterized in that: When the program instructions are executed by the processor, the second electronic device executes the following steps: The plurality of image frames to be inserted are generated according to the marked image frames, and the manipulation operation on the first electronic device includes a manipulation operation in which the manipulation position does not change.
23. The second electronic device according to claim 22, characterized in that: When the program instructions are executed by the processor, the second electronic device executes the following steps: Copying a plurality of copies of the labeled image frame; The multiple image frames to be inserted are obtained according to the multiple copies of the marked image frames.
24. The second electronic device according to claim 21, characterized in that: When the program instructions are executed by the processor, the second electronic device executes the following steps: Generate a plurality of image frames to be annotated according to the image frame corresponding to the input time, wherein the plurality of image frames to be annotated have a sequence; According to the sequence between the multiple control positions and the sequence between the multiple image frames to be annotated, the corresponding control position and the control type of the control operation are respectively annotated in each image frame to be annotated, so as to obtain the multiple image frames to be inserted; the control operation on the first electronic device is a control operation with gradually changing control positions; there are multiple control positions of the control operation in the control parameters of the control operation, and the multiple control positions have a sequence; The earlier the sequence of the marked manipulation positions is, the earlier the sequence of the marked image frames to be marked is; and the later the sequence of the marked manipulation positions is, the later the sequence of the marked image frames to be marked is.
25. The second electronic device according to claim 24, characterized in that: When the program instructions are executed by the processor, the second electronic device executes the following steps: Copying a plurality of copies of the image frame corresponding to the input time; The multiple image frames to be marked are obtained according to the multiple copies of the image frames corresponding to the input time.
26. The second electronic device according to claim 16, characterized in that: When the program instructions are executed by the processor, the second electronic device executes the following steps: receiving the capture time of the projected image frame sent by the first electronic device; storing the identifier of the projected image frame and the capture time in the correspondence between the identifier of the image frame and the capture time, wherein the capture time includes the time when the first electronic device captures the projected image frame; And, taking the input time as the interception time, searching for the image frame corresponding to the input time in the corresponding relationship.
27. A computer storage medium, characterized in that The method comprises computer instructions, which, when executed on a first electronic device, enable the first electronic device to execute the steps of the method executed by the first electronic device as claimed in any one of claims 1 to 12.
28. A computer storage medium, characterized in that The method comprises computer instructions, which, when executed on a second electronic device, enable the second electronic device to execute the steps of the method executed by the second electronic device as claimed in any one of claims 1-11 and 13.
29. A computer program product, characterized in that When the computer program product is executed on a computer, the computer is caused to execute the steps of the method executed by the first electronic device as claimed in any one of claims 1 to 12.
30. A computer program product, characterized in that When the computer program product is executed on a computer, the computer is enabled to execute the steps of the method executed by the second electronic device as claimed in any one of claims 1-11 and 13.
31. A chip, characterized in that: It comprises one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from a memory of a first electronic device and send the signal to the processor, the signal including a computer instruction stored in the memory; when the processor executes the computer instruction, the first electronic device executes the steps of the method executed by the first electronic device as described in any one of claims 1-12.
32. A chip, characterized in that: It comprises one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from a memory of a second electronic device and send the signal to the processor, the signal comprising a computer instruction stored in the memory; when the processor executes the computer instruction, the second electronic device executes the steps of the method executed by the second electronic device as claimed in any one of claims 1-11 and 13.
