Application interaction method, device, electronic device and storage medium

Abstract

The disclosed embodiment discloses an application interaction method, device, electronic device and storage medium, wherein the method includes: in response to the triggering operation of starting a control in a first interface, collecting a target area image, and performing learning content recognition on the target area image; matching an auxiliary tool control based on the recognized learning content, and displaying the matched auxiliary tool control on a second interface. The technical solution of the disclosed embodiment, by displaying matching auxiliary tools on a second interface based on the learning content obtained by real-time recognition, can automatically recommend the most suitable learning auxiliary tools to users, realize intelligent interaction between users and applications, and improve users' learning efficiency and interactive experience.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer technology, and in particular, to an application interaction method, device, electronic device, and storage medium. Background Art

[0002] At present, common learning application (Application, APP) products on the market usually classify learning auxiliary tools based on subject dimensions. For example, English learning APPs may include auxiliary tools required for the English learning process, such as word lookup tools.

[0003] Generally, each learning APP installed on an electronic device can start the auxiliary tool in response to the user's triggering operation on the auxiliary tool contained in it to assist the user in learning. The shortcomings at least include: during the learning process, the user may need to repeatedly trigger and exit the learning auxiliary tools in many APPs to filter out the most suitable auxiliary tool at the moment, resulting in a long learning process, low efficiency, and poor user interaction experience. Summary of the invention

[0004] The embodiments of the present disclosure provide an application interaction method, device, electronic device, and storage medium, which can realize intelligent interaction between users and applications and improve the user's interaction experience.

[0005] In a first aspect, an embodiment of the present disclosure provides an application interaction method, including:

[0006] In response to a triggering operation of a start control in the first interface, collecting a target area image, and performing learning content recognition on the target area image;

[0007] Auxiliary tool controls are matched based on the identified learning content, and the matched auxiliary tool controls are displayed on the second interface.

[0008] In a second aspect, the present disclosure also provides an application interaction device, including:

[0009] An image recognition module, for collecting a target area image in response to a triggering operation of a start control in the first interface, and performing learning content recognition on the target area image;

[0010] The tool matching module is used to match the auxiliary tool control based on the identified learning content and display the matched auxiliary tool control on the second interface.

[0011] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:

[0012] one or more processors;

[0013] a storage device for storing one or more programs,

[0014] When the one or more programs are executed by the one or more processors, the one or more processors implement the application interaction method as described in any one of the embodiments of the present disclosure.

[0015] In a fourth aspect, an embodiment of the present disclosure further provides a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to execute the application interaction method as described in any one of the embodiments of the present disclosure.

[0016] The technical solution of the disclosed embodiment, in response to the triggering operation of the start control in the first interface, collects the target area image, and identifies the learning content of the target area image; matches the auxiliary tool control based on the identified learning content, and displays the matched auxiliary tool control on the second interface. The technical solution of the disclosed embodiment, after the user triggers the start control in the first interface, the application can display the matching auxiliary tools on the second interface based on the learning content obtained by real-time identification, so as to automatically recommend the most suitable learning auxiliary tools to the user, realize the intelligent interaction between the user and the application, and improve the user's learning efficiency and interactive experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0018] Figure 1 A flowchart of an application interaction method provided in the first embodiment of the present disclosure;

[0019] Figure 2 A schematic diagram of a first interface in an application interaction method provided in Embodiment 1 of the present disclosure;

[0020] Figure 3 A schematic diagram of a second interface in an application interaction method provided in the first embodiment of the present disclosure;

[0021] Figure 4 A schematic diagram of a third interface in an application interaction method provided in the first embodiment of the present disclosure;

[0022] Figure 5 A schematic diagram of a second interface in an application interaction method provided in Embodiment 2 of the present disclosure;

[0023] Figure 6 A schematic diagram of a time-consuming editing floating layer in an application interaction method provided in Embodiment 2 of the present disclosure;

[0024] Figure 7 A schematic diagram of a pop-up window interface including an exit prompt in an application interaction method provided in Embodiment 2 of the present disclosure;

[0025] Figure 8 A schematic diagram of a fourth interface in an application interaction method provided in Embodiment 3 of the present disclosure;

[0026] Fig. 9 A schematic diagram of a fifth interface in an application interaction method provided in Embodiment 3 of the present disclosure;

[0027] Fig.10 A schematic diagram of a sixth interface in an application interaction method provided in Embodiment 3 of the present disclosure;

[0028] Fig.11 A schematic diagram of a seventh interface in an application interaction method provided in Embodiment 3 of the present disclosure;

[0029] Fig.12 A schematic diagram of an eighth interface in an application interaction method provided in Embodiment 4 of the present disclosure;

[0030] Fig.13 A schematic diagram of a ninth interface in an application interaction method provided in Embodiment 4 of the present disclosure;

[0031] Fig.14 A schematic diagram of a pop-up window interface including a sending progress control in an application interaction method provided in a fourth embodiment of the present disclosure;

[0032] Fig.15 A schematic diagram of the tenth interface in an application interaction method provided in the fourth embodiment of the present disclosure;

[0033] Fig.16 A schematic diagram of the structure of an application interaction device provided in Embodiment 5 of the present disclosure;

[0034] Fig.17 This is a schematic diagram of the structure of an electronic device provided in Embodiment 6 of the present disclosure. DETAILED DESCRIPTION

[0035] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0036] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0037] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0038] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0039] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0040] Embodiment 1

[0041] Figure 1 This is a flow chart of an application interaction method provided in the first embodiment of the present disclosure. The embodiment of the present disclosure is applicable to the situation where students interact with a homework application when using the homework application to complete their homework. The method can be executed by an application interaction device, which can be implemented in the form of software and / or hardware. The device can be integrated into a client application and can be installed in an electronic device along with the application, such as a mobile phone, tablet computer, laptop computer, or desktop computer.

[0042] like Figure 1 As shown, the application interaction method provided in this embodiment includes:

[0043] S110 . In response to a triggering operation of a start control in the first interface, a target area image is collected, and learning content recognition is performed on the target area image.

[0044] The application (Application, APP) provided in the embodiments of the present disclosure can be considered as a learning application, for example, it can be a homework application that assists students in doing homework in various subjects. When the application is installed in an electronic device such as a smart phone or a computer, the electronic device can start the application in response to a user start operation, for example, the electronic device can start the application in response to a user clicking on an application icon.

[0045] When the application is started, the first interface can be used as the home page interface of the application, or as a sub-level interface of the home page interface, and a start control can be displayed in the first interface. When the user is ready to start learning, the start control can be triggered so that the application can intelligently recommend learning auxiliary tools according to the real-time situation of the user's learning process to achieve intelligent interaction.

[0046] In some optional implementations, the application may establish a communication connection with at least one acquisition device in advance, and the acquisition device may include but is not limited to: image acquisition devices such as cameras, audio acquisition devices such as microphones, human body signs monitoring devices such as body temperature, and environmental parameter acquisition devices such as ambient light, etc. In these optional implementations, by establishing a communication connection between the application and the acquisition device, the application can receive various parameters such as images, audio and video, physical sign parameters, and environmental parameters collected by the acquisition device during the user's learning process. By analyzing various parameters, it can provide a full range of services for the user's learning process, such as recording the learning process, prompting the learning status, prompting the health status, and prompting the environmental status.

[0047] In this embodiment, the application can establish a communication connection with the image acquisition device in advance, and when the application detects a trigger operation acting on the start control in the first interface, it can control the image acquisition device to acquire images in response to the trigger operation. The target area image can be an image of the acquisition area of ​​the image acquisition device, or it can be an area image obtained by preprocessing the image of the acquisition area. The preprocessing of the image of the acquisition area can include but is not limited to: cropping of the book area, rectangular correction processing to remove distortion, and / or rotation correction processing to adjust the image angle. In order to enable the acquired target area image to better present the learning process, the image acquisition device can be an overhead image acquisition device deployed on a desk desktop that can take a bird's-eye view, such as a top-mounted camera of a smart desk lamp.

[0048] The learning content may include but is not limited to subjects and tasks. Subjects may be textbook subjects, such as Chinese, mathematics, English, physics, geography, etc., and tasks may be the names of books to which homework tasks belong, such as oral arithmetic exercise books, dictation homework books, winter vacation homework, etc. By recognizing the target area image, learning content such as subjects, subjects and tasks can be parsed.

[0049] The process of learning content recognition for the target area image may be to directly recognize the learning content based on the target area image, or to first recognize the target area image for text recognition, and then recognize the learning content based on the recognized text. Furthermore, the target area image to be recognized may be a target area image captured in real time, or may be a preset number of target area images screened from the target area images captured in real time. The method of screening the target area image may be to obtain the currently captured target area image at regular intervals, and other screening methods may also be applied here.

[0050] By identifying the target area images collected in real time, it is helpful to intelligently recommend the most suitable learning auxiliary tools according to the subject and / or task information of the user in the learning process, thereby improving the user experience. In addition, by identifying the selected target area images, it is possible to reduce the processing resources consumed by the application and improve the fluency of the application on the basis of meeting the needs of real-time recommendation of auxiliary tools to a certain extent.

[0051] S120: Match auxiliary tool controls based on the identified learning content, and display the matched auxiliary tool controls on the second interface.

[0052] In the disclosed embodiment, the correspondence between learning content and auxiliary tools can be established in advance, for example, the correspondence between subjects and auxiliary tools can be established in advance; for example, the correspondence between subjects and tasks and auxiliary tools can be established in advance. For example, the correspondence between the subject "Chinese" and auxiliary tools such as "looking up a dictionary" and "reading a text" can be established in advance, and the correspondence between the subject "Chinese" and the task of "dictation workbook" and the auxiliary tool "reading a text" can also be established in advance.

[0053] Generally speaking, for a certain science subject, establishing a correspondence between subjects and tasks and auxiliary tools is more conducive to matching a smaller range of more accurate auxiliary tools than establishing a correspondence between subjects and auxiliary tools. In some cases where the task cannot be accurately identified, the automatic recommendation of auxiliary tools can still be guaranteed by establishing a correspondence between subjects and auxiliary tools.

[0054] Correspondingly, matching the auxiliary tool controls based on the identified learning content can be based on a pre-established correspondence, determining the auxiliary tool corresponding to the identified learning content, and obtaining the corresponding auxiliary tool controls to achieve automatic matching of the auxiliary tool controls.

[0055] When learning content is identified, or when an auxiliary tool control is matched, the first interface can be jumped to the second interface. The timing of the jump is not specifically limited here. When jumping to the second interface, the auxiliary tool control can be added to the entry position reserved for the second interface to display the auxiliary tool control in the second interface. By triggering the auxiliary tool control, relevant auxiliary functions can be implemented to assist users in learning, avoiding the time-consuming process of users screening suitable auxiliary tools, and improving learning efficiency and interactive experience.

[0056] Among them, when the number of matching auxiliary tool controls is large, and the display entry positions of the second interface are small, some auxiliary tool controls can also be added to the hidden entry positions of the second interface. In addition, in response to user operations in an area (such as a toolbox or toolbar area) where auxiliary tool controls are collected, the hidden auxiliary tool controls can be displayed in sequence, and the displayed auxiliary tool controls can be hidden in sequence, so as to realize the display of each matching auxiliary tool control.

[0057] In some optional embodiments, learning content recognition is performed on the target area image, including: text recognition is performed on the target area image, and the subject corresponding to the recognized text is determined; accordingly, auxiliary tool controls are matched based on the recognized learning content, including: auxiliary tool controls are matched based on the determined subject.

[0058] Among them, an optical character recognition (OCR) method and / or an OpenCV text recognition and detection method may be used to perform text recognition on the target area image, and other text recognition methods may also be applied here.

[0059] Among them, determining the subject corresponding to the recognized text can be by inputting the recognized text into a pre-trained model, and outputting the subject corresponding to the recognized text based on the pre-trained model, and the pre-trained model can be, for example, a classification model such as a neural network model; or the recognized text can be matched with text data corresponding to the learning content in a preset database, and the subject can be determined based on the matching result.

[0060] Among them, matching the auxiliary tool controls based on the determined disciplines may be based on a pre-established correspondence between disciplines and auxiliary tools, determining the auxiliary tools corresponding to the identified disciplines, and obtaining the corresponding auxiliary tool controls to achieve automatic matching of the auxiliary tool controls.

[0061] For example, Figure 2 This is a schematic diagram of a first interface in an application interaction method provided in Embodiment 1 of the present disclosure.

[0062] See also Figure 2, the application to which the first interface belongs may be an application for assisting users in doing homework. The first interface of the application may display a start control 10, wherein the start control may be a button control or other types of controls, and a prompt text such as "start doing homework" may be added to the start control to prompt the user to trigger the start control when he is ready to start learning. And the application may respond to the triggering operation of the start control 10 in the first interface, collect the target area image, and perform learning content recognition on the target area image.

[0063] In addition, the first interface can also present a schematic diagram of the camera capturing the desk desktop image, to prompt the user to place the learning materials within the camera capture range to ensure smooth recognition of the learning content.

[0064] For example, Figure 3 This is a schematic diagram of a second interface in an application interaction method provided in Embodiment 1 of the present disclosure.

[0065] See also Figure 3 In the second interface, a homework toolbox 201 can be set, and the homework toolbox 201 can be considered as an area for collecting auxiliary tool controls. Multiple entry positions can be set in the homework toolbox 201, and auxiliary tool controls obtained by matching the identified learning content can be added to the entry positions, such as "reading text" and "looking up dictionary" tool controls. In addition, default auxiliary tool controls can also be added to the entry positions, such as the "photographing" tool control.

[0066] like Figure 3 As shown, when the number of matching auxiliary tool controls is large, and the display entry positions of the second interface are small, multiple pages can be set in the work toolbox 201 according to the number of auxiliary tool controls, and a default display page and a default hidden page can be set, and auxiliary tool controls can be added at the entry position in each page. Figure 3 The homework toolbox 201 is provided with three pages, and the middle page can be a default displayed page, and the left and right pages can be default hidden pages. Three auxiliary tool controls, "read text", "check dictionary" and "take photo", can be added to the entrance position of the middle page, and other auxiliary tool controls can be added to the entrance positions of the left and right pages.

[0067] Furthermore, the application can switch the currently displayed page in response to the left and right sliding operation of the job toolbox 201 to display the hidden auxiliary tool controls, and hide the displayed auxiliary tool controls at the same time, thereby realizing the sequential display of the auxiliary tool controls.

[0068] In some optional implementations, the process of performing learning content recognition on the target area image further includes: displaying the target area image on a third interface, and rendering a recognition animation on the displayed target area image.

[0069] Among them, if the process of learning content recognition takes a long time, it is easy for the first interface to take a long time to jump to the second interface, and the user cannot perceive whether content recognition is currently being performed, which affects the user experience to a certain extent.

[0070] In these optional implementations, during the learning content recognition process, the first interface can be jumped to the third interface, the real-time acquired target area image can be displayed in the third interface, and the recognition animation can be rendered on the target area image, so as to prompt the user that the application is currently recognizing the learning content, so as to improve the user's interactive experience. In addition, when the learning content is successfully recognized, the third interface can be jumped to the second interface to display the matching auxiliary tools.

[0071] For example, Figure 4 This is a schematic diagram of a third interface in an application interaction method provided in Embodiment 1 of the present disclosure.

[0072] like Figure 4 As shown, the display control 301 in the third interface can be used to display the target area image collected in real time, and the recognition animation of the scan line control 302 moving up and down can be rendered on the target area image to prompt the user that the learning content recognition is being performed on the target area image.

[0073] See also Figure 4 , the third interface may also include but is not limited to: prompt text control 303, default auxiliary tool control 304, completion control 305 and exit control 306. Among them, the prompt text control 303 can be used to display the prompt text, and the prompt text is, for example, "Open the book and start doing homework", "After identifying the book, we will recommend auxiliary tools for you", etc.; the default auxiliary tool control 304 can be used to start the default auxiliary function when triggered, and the default auxiliary control is, for example, "recording", "video recording" and "photographing" and other auxiliary controls that are not matched by the learning content; the completion control 305 can be added with text such as "finishing homework", which can be used to determine the completion of this learning process when triggered; the exit control 306 can be used to exit the first interface when triggered.

[0074] The technical solution of the disclosed embodiment, in response to the triggering operation of the start control in the first interface, collects the target area image, and identifies the learning content of the target area image; matches the auxiliary tool control based on the identified learning content, and displays the matched auxiliary tool control on the second interface. After the user triggers the start control in the first interface, the application can display the matching auxiliary tool on the second interface based on the learning content obtained by real-time identification, so as to automatically recommend the most suitable learning auxiliary tool to the user, realize the intelligent interaction between the user and the application, and improve the user's learning efficiency and interactive experience.

[0075] Embodiment 2

[0076] This embodiment can be combined with the various optional schemes in the application interaction method provided in the above embodiments. The application interaction method provided in this embodiment optimizes the interactive steps of recording the learning process, and can record the target area image according to preset rules, and / or record the target area image, audio data and / or video data through interaction with relevant controls in the second interface, so as to record the learning process in the process of completing the learning content, and further improve the user interaction experience.

[0077] In some optional implementations, the process of matching auxiliary tool controls based on the identified learning content also includes: recording the target area image according to preset rules; and / or, in response to the trigger operation of the photo control in the second interface, recording the currently captured target area image; and / or, in response to the trigger operation of the recording control in the second interface, recording audio data; and / or, in response to the trigger operation of the video recording control in the second interface, recording video data corresponding to the target area image; generating a recording control based on the recording results, and displaying the recording control on the second interface.

[0078] The traditional way of recording learning process data usually requires manual recording, which reduces learning efficiency to a certain extent. In these optional implementations of the embodiment, the application can automatically record the target area image data based on preset rules during the process of matching the auxiliary tool control, which can improve the user's learning efficiency to a certain extent.

[0079] Among them, a photo control can also be set on the second interface, so that when the automatic recording of the target area image fails, the application can respond to the trigger operation of the photo control to record the currently collected target area image, thereby improving the fault tolerance of the application.

[0080] In addition, in addition to completing written learning, the learning process can also complete non-written learning such as reading aloud and reciting. Therefore, in these optional implementations, the second interface can also be provided with a recording control and a video control, so that the application can respond to the triggering operation of the recording control and the video control to record the learning process of non-written learning. Among them, recording the video data corresponding to the target area image can be considered as recording the target area image collected in real time, and while recording the target area image collected in real time, the audio data can also be recorded.

[0081] Among them, the photo taking control, the audio recording control and the video recording control can be considered as default auxiliary tool controls, and can be set in the same area of ​​the second interface as the matched auxiliary tool controls.

[0082] Among them, generating a record control according to the record result can be generating a corner control according to the total number of recorded target area images, audio data and video data, and setting the corner control in the record control so that the user can directly understand the total number of data currently recorded. In addition, when the recorded data is empty, that is, when no target area image, audio data and video data are recorded, the second interface can also not display the record control first, so that the user can clearly understand the current recording of the learning process.

[0083] In some optional implementations, the target area image is recorded according to preset rules, including: performing learning content recognition on the currently acquired target area image and / or performing clarity assessment on the currently acquired target area image at preset time intervals; when the recognized learning content and / or clarity assessment meets the standards, the currently acquired target area image is recorded.

[0084] In these optional implementations, the learning content and / or clarity of the currently acquired target area image can be evaluated at preset time intervals (e.g., 1s, 3s, etc.), and high-quality images that contain learning content and are clear can be recorded, which is conducive to capturing key images and improving the accuracy of recording learning process data.

[0085] For example, Figure 5 A schematic diagram of a second interface in an application interaction method provided in Embodiment 2 of the present disclosure.

[0086] See also Figure 5 The second interface may include a work toolbox 201. In addition to the matching auxiliary tool controls and the default auxiliary tool controls, the work toolbox may also include a record control 202. In addition, the record control 202 may include a corner mark control representing the total number of data currently recorded. Figure 5 The style of the superscript control is a solid circle with numbers, and it can be other styles.

[0087] See again Figure 5 In some optional implementations, the second interface also displays at least one of the following controls: a learning content prompt control 203, a planned time prompt control 204, and an actual time prompt control 205; accordingly, the application interaction method also includes: in response to the triggering operation of the planned time prompt control 204 in the second interface, editing the planned time.

[0088] Among them, the learning content prompt control 203 can be used to prompt the currently recognized learning content, such as subject and / or task information. Among them, the actual time prompt control 205 can be used to prompt the user of the actual time spent on the current subject, or the current subject and the current task, and when a new subject and / or task is recognized, the timing can be reset. Among them, the planned time prompt control 204 can be used to display the edited planned time; and when the planned time is not edited, the application can respond to the triggering operation of the planned time prompt control 204 and jump to the editing floating layer to edit the planned time.

[0089] For example, Figure 6 This is a schematic diagram of a time-consuming editing floating layer in an application interaction method provided in the second embodiment of the present disclosure. Figure 6 The editing pop-up layer 2041 may include but is not limited to: a time value input control to input a value of the planned time consumption; a confirmation control to confirm that the editing of the planned time consumption is completed; and an exit control to exit the editing pop-up layer 2041 .

[0090] See also Figure 5 The second interface may also include, but is not limited to: an exit control 206, a completion control 207, and a display control 208. The exit control 206 may be used to exit the second interface when triggered; the completion control 207 may be added with text such as "finished homework" and may be used to determine the completion of this learning process when triggered; the display control 208 may be used to display the target area image collected in real time. When the exit control in the second interface or the exit control in the third interface is triggered, a pop-up window interface may also be popped up to provide an exit prompt, so as to determine whether the user really needs to exit and avoid exiting due to accidental touch.

[0091] For example, Figure 7 A schematic diagram of a pop-up window interface including an exit prompt in an application interaction method provided in Embodiment 2 of the present disclosure.

[0092] See also Figure 7, which is a pop-up window interface 2061 that pops up when the exit control in the second interface is triggered. A mask layer with a preset transparency can be covered on the second interface, and the pop-up window interface 2061 can be set on the mask layer. The pop-up window interface 2061 can include but is not limited to: a prompt text control, which can be used to display a prompt text, and the prompt text is, for example, "The homework has not been completed yet, are you sure to exit?"; an exit control, which is used to confirm exiting the second interface when triggered; a close pop-up window control or a complete homework control, which is used to return to the second interface to continue to provide auxiliary tool control recommendations for the user's learning process and other functions.

[0093] By providing prompts on learning content, planned time, actual time and other information, users can understand more details of the learning process and further improve the user experience.

[0094] The technical solution of the embodiment of the present disclosure optimizes the interactive steps of recording the learning process, and can record the target area image according to preset rules, and / or record the target area image, audio data and / or video data through interaction with relevant controls in the second interface, so as to record the learning process in the process of completing the learning content, and further improve the user interaction experience. In addition, the application interaction method provided by the embodiment of the present disclosure and the application interaction method provided by the above embodiment belong to the same technical concept, and the technical details not described in detail in this embodiment can be referred to the above embodiment, and the same technical features have the same beneficial effects in this embodiment and the above embodiment.

[0095] Embodiment 3

[0096] This embodiment can be combined with the various optional solutions in the application interaction method provided in the above embodiment. The application interaction method provided in this embodiment optimizes the interaction steps recorded in the viewing learning process, and can realize the viewing of the recorded target area image, audio data and / or video data, further improving the user interaction experience.

[0097] In some optional implementations, the application interaction method also includes: in response to a trigger operation of a recording control in the second interface, displaying a thumbnail of the recorded target area image, a cover image of the audio data, and / or a cover image of the video data on a fourth interface.

[0098] For example, Figure 8 This is a schematic diagram of the fourth interface in an application interaction method provided in the third embodiment of the present disclosure. Figure 8 The fourth interface may include, but is not limited to: an area 401 containing information related to the target area image, an area 402 containing information related to audio and video, an area 403 containing default auxiliary tool controls, and a return control 404.

[0099] Among them, area 401 may include but is not limited to: a prompt text control, which can be used to display the prompt text, and the prompt text may include, the easy-to-understand text "written homework" corresponding to the "target area image", and the value "4" used to indicate the number of target area images; a display control of the target area image, which can be used to display a thumbnail of the target area image recorded according to preset rules or in response to the camera control of the second interface, and the display control can be set with a superscript control containing learning content, such as setting the superscript controls of the subjects "Chinese" and "Mathematics".

[0100] Among them, area 402 may include but is not limited to: a prompt text control, which can be used to display the prompt text, and the prompt text may include, an easy-to-understand text "audio and video operation" summarizing "audio data" and "video data", and a value "1" used to prompt the total number of audio data and video data; a cover image display control, which can be used to display the cover image of audio data or video data, and the cover image of audio data can be a fixed preset image, and the cover image of video data can be a thumbnail of an image of a certain frame of video data.

[0101] Among them, area 403 may include but is not limited to default auxiliary tool controls such as "photograph", "video record" and / or "audio record" to enrich the entry position of the default auxiliary tool, so that the default auxiliary tool controls can be triggered more flexibly, which is convenient for users to record the learning process. Among them, the return control 404 can be used to return to the second interface when triggered.

[0102] In these optional implementations, a thumbnail of the recorded target area image, a cover image of the audio data, and / or a cover image of the video data can be displayed in the fourth interface, which can help users understand the specific circumstances of recording the learning process, thereby improving the user experience.

[0103] In some optional implementations, if the fourth interface displays a thumbnail of the target area image, it also includes: in response to a triggering operation of the thumbnail, displaying the target area image corresponding to the triggered thumbnail on the fifth interface; in response to a viewing operation received on the fifth interface, performing a corresponding viewing action on the currently displayed target area image.

[0104] When the thumbnail of the target area image displayed on the fourth interface is triggered, that is, when the display control of the thumbnail is triggered, the original image of the target area image corresponding to the triggered thumbnail can be displayed on the fifth interface. The viewing operations received by the fifth interface include but are not limited to zooming in, zooming out, switching images, and dragging; the corresponding viewing actions performed on the target area image include but are not limited to zooming in on the target area image, zooming out the target area image, switching to other target area images, and dragging the target area image.

[0105] For example, Fig. 9 This is a schematic diagram of the fifth interface in an application interaction method provided in Embodiment 3 of the present disclosure.

[0106] See also Fig. 9 The fifth interface may include but is not limited to: an original image display control 501, which can be used to display the original image of the target area image corresponding to the triggered thumbnail; a delete control 502, which can be used to delete the currently displayed target area image; a prompt text control 503, which can be used to prompt that the current target area image is located at the position of all recorded target area images, for example, "1 / 4" can represent that there are 4 target area images in all records, and the current target area image is the first one; an exit control 504, which can be used to exit the fifth interface.

[0107] In some optional implementations, if the fourth interface displays a cover image of audio data, it also includes: in response to a triggering operation of the cover image of the audio data, playing the audio data corresponding to the triggered cover image, and displaying relevant information of the played audio data on the sixth interface; in response to a playback parameter control operation received on the sixth interface, performing playback parameter control on the currently playing audio data.

[0108] In some optional implementations, if the fourth interface displays a cover image of video data, it also includes: in response to a triggering operation of the cover image of the video data, playing the video data corresponding to the triggered cover image in the seventh interface; in response to a playback parameter control operation received on the seventh interface, controlling the playback parameters of the currently playing video data.

[0109] Among them, by triggering the cover image of audio data or video data, the audio data and video data can be played accordingly, and the playback interfaces of the two can be the sixth interface and the seventh interface respectively.

[0110] Among them, the playback parameter control operations received by the sixth interface include but are not limited to volume parameter control operations, pause / play control operations, switching control operations and playback mode setting operations; accordingly, the playback parameter control of the currently playing audio data is performed, including but not limited to adjusting the volume, controlling play or pause, switching audio data and setting the playback mode (for example, single play, repeated play, etc.).

[0111] Among them, the playback parameter control operations received by the seventh interface include but are not limited to brightness parameter control operations, volume parameter control operations, pause / play control operations, switching control operations and playback mode setting operations; accordingly, the playback parameter control of the currently playing video data is performed, including but not limited to adjusting the video brightness, adjusting the volume, controlling playback or pause, switching video data and setting the playback mode (for example, single playback, repeated playback, etc.).

[0112] For example, Fig.10 This is a schematic diagram of the sixth interface in an application interaction method provided in Embodiment 3 of the present disclosure.

[0113] See also Fig.10 , the sixth interface may include but is not limited to: a cover image display control 601, which can be used to display the cover image of the audio data, and the cover image display control can also be set to a bottom-level control of the same size as the sixth interface, so that the cover image can be used as the background image of the sixth interface; a prompt text control 602, which can be used to prompt the name of the currently playing audio data and other information; a progress bar control 603, which can be used to prompt the progress of the current playback, and the left side can display the duration of the currently played audio data, and the right side can display the total duration of the audio data; a playback parameter control 604, which is used to control the playback parameters of the currently playing audio data when triggered, and the figure only shows a switch control (such as a solid circle plus a vertical line triangle control) and a playback control (such as a solid circle plus a triangle control), and other controls may also be included here; a delete control 605, which can be used to delete the currently playing audio data; an exit control 606, which can be used to exit the sixth interface.

[0114] For example, Fig.11 This is a schematic diagram of the seventh interface in an application interaction method provided in Embodiment 3 of the present disclosure.

[0115] See also Fig.11 The seventh interface may include but is not limited to: a video data playback control 701, which can be used to play video data; a playback parameter control 702, which is used to control the playback parameters of the currently playing video data when triggered. The figure only shows the playback control (for example, a solid circle plus a triangle style control), and other controls may also be included here, and the playback parameter control 702 may be hidden during video playback; a delete control 605, which can be used to delete the currently playing video data; an exit control 606, which can be used to exit the seventh interface.

[0116] Among them, the seventh interface can be used as the sixth interface to play the audio data, and at this time, the playback control of the video data can be used to display the cover image of the audio data.

[0117] The technical solution of the embodiment of the present disclosure optimizes the interactive steps recorded in the viewing learning process, and can realize the viewing of the recorded target area image, audio data and / or video data, further improving the user interaction experience. In addition, the application interaction method provided by the embodiment of the present disclosure and the application interaction method provided by the above embodiment belong to the same technical concept, and the technical details not fully described in this embodiment can be referred to the above embodiment, and the same technical features have the same beneficial effects in this embodiment and the above embodiment.

[0118] Embodiment 4

[0119] This embodiment can be combined with the various optional solutions in the application interaction method provided in the above embodiment. The application interaction method provided in this embodiment optimizes the interaction steps recorded in the sending learning process, and can realize the sending of the recorded target area image, audio data and / or video data, further improving the user interaction experience.

[0120] In some optional implementations, the application interaction method also includes: in response to the triggering operation of the completion control in the second interface, determining whether the target area image, audio data and / or video data are currently recorded; if so, displaying the thumbnail of the target area image in the editing state, the cover image of the audio data, and / or the cover image of the video data on the eighth interface; in response to the triggering operation of the send control in the eighth interface, sending the target area image corresponding to the thumbnail selected in the editing state, the audio data corresponding to the cover image of the audio data, and / or the video data corresponding to the cover image of the video data to the account of the target application.

[0121] When the completion control in the second interface or the completion control in the third interface is triggered, it can be determined that the learning process is completed, and the step of sending the data recorded in the learning process can also be triggered.

[0122] The step of sending the recorded data of the learning process may include: in response to the selection operation of the thumbnail and / or cover image in the edit state in the eighth interface, determining the target area image, audio data and / or video data to be sent; in response to the triggering operation of the send control in the eighth interface, sending the data to be sent to the account of the target application. The target application may be the application provided by the embodiment of the present disclosure, or other applications that can communicate with the application provided by the embodiment of the present disclosure, such as an email application or an instant messaging application.

[0123] In these optional implementations, by sending the recorded data of the selected learning process to the account of the target application, it is possible to send the data of the student's learning process to the teacher's or parent's account of the target application, so that the teacher or parent can supervise the student's learning process and correct homework, etc.

[0124] For example, Fig.12 This is a schematic diagram of the eighth interface in an application interaction method provided in Embodiment 4 of the present disclosure.

[0125] See also Fig.12The eighth interface can be considered as the fourth interface in the editing state, and may also include some areas and controls in the fourth interface, such as area 801 containing information related to the target area image, area 802 containing audio and video related information, area 804 containing default auxiliary tool controls, and return control 805. The same technical details can be found above and will not be repeated here.

[0126] In addition, the difference between the eighth interface and the fourth interface may include that the display control of the thumbnail of the target area image and the display control of the cover image of the audio data or video data in the eighth interface can be edited by selecting or not selecting. Fig.12 In the example, the upper left corner of the selected display control may be marked with a solid circle plus a check mark, and the upper left corner of the unselected display control may be marked with a hollow circle.

[0127] At the same time, the prompt text control in the eighth interface can be updated according to the number of selected data. Fig.12 In the example, when there are a total of 4 target area images recorded in the written assignment and 3 target area data are currently selected, the prompt text control can be updated to "written assignment 3 / 4".

[0128] When the user selects all the data to be sent, the sending control in the eighth interface can be triggered, for example Fig.12 The application can respond to the triggering operation of the send control in the eighth interface and send the selected data to the account of the target application.

[0129] In some optional implementations, the application interaction method also includes: if the target area image, audio data and / or video data are not currently recorded, displaying the photo control, audio recording control and / or video recording control on the ninth interface; in response to the triggering operation of the photo control in the ninth interface, recording the currently captured target area image; and / or, in response to the triggering operation of the audio recording control in the ninth interface, recording the audio data; and / or, in response to the triggering operation of the video recording control in the ninth interface, recording the video data corresponding to the target area image; displaying the thumbnail of the recorded target area image in the editing state, the cover image of the audio data, and / or the cover image of the video data on the eighth interface.

[0130] Among them, if the application fails to automatically record the target area image, and the user does not trigger the default auxiliary tool controls such as "photograph", "record" or "record" in the second interface to record the learning process, the current recorded data is empty, that is, no target area image, audio data and / or video data are currently recorded. At this time, if the completion control in the second interface or the completion control in the third interface is triggered, the user can be prompted to record the learning process first, and then the step of sending the recorded data in the learning process is triggered.

[0131] In these optional implementations, when the completion control in the second interface or the third interface is triggered and the current recorded data is empty, the ninth interface can be used to display default auxiliary tool controls such as "photograph", "record" or "video", so as to assist the user in recording the learning process more conveniently. And when the user finishes recording, he can jump to the eighth interface to send the recorded data.

[0132] For example, Fig.13 This is a schematic diagram of the ninth interface in an application interaction method provided in Embodiment 4 of the present disclosure.

[0133] See also Fig.13 The ninth interface may include but is not limited to: an area 901 including a default auxiliary tool control, a send control 902, a prompt text control 903 and a return control 904.

[0134] The default auxiliary tools included in area 901 include, but are not limited to, a photo control, a sound recording control, and a video recording control, and each control may be provided with a function prompt text, such as Fig.13 A reminder text such as "Take a photo of each page of your homework" can be added to the photo control to remind the user of the functional scenarios of each control. Among them, when currently recording data, the send control 902 can be in a disabled state and can be displayed differently from the send control in an enabled state, for example Fig.13 The send control in the disabled state may be dark. Among them, the prompt text control 903 can add prompt text, for example, the prompt text control in 13 can add the prompt text "Record homework" to prompt the user to record the learning process. Among them, the return control 904 can be used to return to the second interface or the third interface when triggered.

[0135] In some optional implementations, the process of sending the target area image corresponding to the thumbnail selected in the editing state, the audio data corresponding to the cover image of the audio data, and / or the video data corresponding to the cover image of the video data to the account of the target application also includes: displaying the sending progress control in a pop-up interface; terminating the sending process in response to the triggering operation of canceling the sending control in the pop-up interface; when the sending is successful, switching the pop-up interface to a sending success prompt control.

[0136] In these optional implementations, by using a pop-up interface to provide a sending progress prompt during the sending process of the data to be sent, and switching the pop-up control to a sending success prompt control after successful sending, the user can clearly understand the progress of data sending, further improving the user experience.

[0137] For example, Fig.14A schematic diagram of a pop-up window interface including a sending progress control in an application interaction method provided in Embodiment 4 of the present disclosure.

[0138] See also Fig.14 , a mask layer with a preset transparency can be covered on the eighth interface, and a pop-up window interface 8031 ​​including a sending progress control can be set on the mask layer. The pop-up window interface 8031 ​​including the sending progress control can include but is not limited to: a sending progress control, which is used to prompt the sending progress, and the sending progress control is, for example, a circular progress bar plus a progress percentage style control, or a control of other styles; a prompt text control, which can add a prompt text, such as adding a text of "sending job", to prompt that data is currently being sent; and a cancel sending control, which is used to cancel the sending of job data when triggered.

[0139] See also Fig.14 When the data transmission is completed, the pop-up window interface 8031 ​​including the transmission progress control can be switched to the transmission success prompt control 8032, and the transmission success prompt control is Fig.14 In addition to the style, there can be other styles, which are not exhaustive here.

[0140] In some optional implementations, the application interaction method further includes: in response to a triggering operation of completing a control in the second interface, determining an actual time consumed to complete the learning content, and displaying the determined actual time consumed on a tenth interface.

[0141] Among them, when the completion control in the second interface or the completion control in the third interface is triggered, it can be determined that the learning process is completed, and the statistical display step of the time consumption of the learning process can also be triggered.

[0142] Among them, the analysis and display step of the time consumption of the learning process may include: obtaining the actual time consumption of each learning content in the learning process to determine the total time consumption of the learning process; and displaying the total time consumption and the actual time consumption of each learning content in the tenth interface. Among them, after the completion control in the second interface is triggered, the second interface can jump to the tenth interface; or after the completion control in the second interface is triggered, the eighth interface is used to send the recorded data, and when the sending is successful, the eighth interface with the sending success prompt control displayed can jump to the tenth interface.

[0143] In these optional implementations, by using the tenth interface to display the actual time consumption, it can help users understand the time consumption of this learning process and further improve the user experience.

[0144] For example, Fig.15 This is a schematic diagram of the tenth interface in an application interaction method provided in Embodiment 4 of the present disclosure.

[0145] See also Fig.15 The tenth interface may include but is not limited to: a display area 1001 containing time-consuming information, in which the time consumed for each learning content and the total time consumed for this learning process can be displayed, and rewards (such as energy rewards, small red flower rewards, etc.) can also be given according to the length of the total time consumed; a user feedback control 1002, which can be used to receive status feedback from the user after completing the current learning process, such as but not limited to "struggle", "happy" and "keep going", etc.; an exit control 1003, which can be used to exit the application when triggered.

[0146] The technical solution of the embodiment of the present disclosure optimizes the interactive steps recorded in the sending learning process, and can realize the sending of the recorded target area image, audio data and / or video data, further improving the user interaction experience. In addition, the application interaction method provided by the embodiment of the present disclosure and the application interaction method provided by the above embodiment belong to the same technical concept, and the technical details not fully described in this embodiment can be referred to the above embodiment, and the same technical features have the same beneficial effects in this embodiment and the above embodiment.

[0147] Embodiment 5

[0148] Fig.16 This is a schematic diagram of the structure of an application interaction device provided in Embodiment 5 of the present disclosure. The application interaction device provided in this embodiment is suitable for students interacting with a homework application when completing their homework using the homework application.

[0149] like Fig.16 As shown, the application interaction device includes:

[0150] The image recognition module 161 is used to collect the target area image and perform learning content recognition on the target area image in response to the triggering operation of the start control in the first interface;

[0151] The tool matching module 162 is used to match auxiliary tool controls based on the identified learning content and display the matched auxiliary tool controls on the second interface.

[0152] In some optional implementations, the image recognition module is specifically used to perform text recognition on the target area image and determine the subject corresponding to the recognized text;

[0153] Correspondingly, the tool matching module is specifically used to match auxiliary tool controls based on the determined disciplines.

[0154] In some optional implementations, the image recognition module is further used to: during the process of performing learning content recognition on the target area image, display the target area image on the third interface, and render a recognition animation on the displayed target area image.

[0155] In some optional implementations, the application interaction device further includes:

[0156] A recording module is used to record the target area image according to preset rules in the process of matching the auxiliary tool control based on the identified learning content; and / or, in response to the triggering operation of the photo control in the second interface, record the currently collected target area image; and / or, in response to the triggering operation of the recording control in the second interface, record the audio data; and / or, in response to the triggering operation of the video recording control in the second interface, record the video data corresponding to the target area image; generate a recording control according to the recording result, and display the recording control on the second interface.

[0157] In some optional implementations, the recording module includes:

[0158] The automatic recording submodule is used to identify the learning content of the currently acquired target area image and / or perform clarity assessment on the currently acquired target area image at preset time intervals; when the identified learning content and / or clarity assessment meets the standards, the currently acquired target area image is recorded.

[0159] In some optional implementations, the application interaction device further includes:

[0160] The record display module is used to display the thumbnail of the recorded target area image, the cover image of the audio data, and / or the cover image of the video data on the fourth interface in response to the triggering operation of the record control in the second interface.

[0161] In some optional implementations, the application interaction device further includes:

[0162] The recording and viewing module is used for displaying the target area image corresponding to the triggered thumbnail on the fifth interface in response to the triggering operation of the thumbnail if the fourth interface displays a thumbnail of the target area image; and performing a corresponding viewing action on the currently displayed target area image in response to the viewing operation received on the fifth interface.

[0163] In some optional implementations, the application interaction device further includes:

[0164] A recording and playback module is used to respond to a triggering operation of the cover image of the audio data if a cover image of the audio data is displayed on the fourth interface, play the audio data corresponding to the triggered cover image, and display the relevant information of the played audio data on the sixth interface; respond to a playback parameter control operation received on the sixth interface, and control the playback parameters of the currently playing audio data.

[0165] In some optional implementations, the recording and playback module is also used to, if the fourth interface displays a cover image of the video data, respond to a triggering operation of the cover image of the video data, play the video data corresponding to the triggered cover image in the seventh interface; and respond to a playback parameter control operation received on the seventh interface to control the playback parameters of the currently playing video data.

[0166] In some optional implementations, the application interaction device further includes:

[0167] A record sending module is used to respond to the trigger operation of the completion control in the second interface, and determine whether the target area image, audio data and / or video data are currently recorded; if so, the thumbnail of the target area image in the editing state, the cover image of the audio data, and / or the cover image of the video data are displayed on the eighth interface; in response to the trigger operation of the send control in the eighth interface, the target area image corresponding to the thumbnail selected in the editing state, the audio data corresponding to the cover image of the audio data, and / or the video data corresponding to the cover image of the video data are sent to the account of the target application.

[0168] In some optional implementations, if the recording and sending module determines that no target area image, audio data and / or video data are currently recorded, the recording module is further used to display a photo control, a recording control and / or a video recording control on the ninth interface; in response to a trigger operation of the photo control in the ninth interface, the currently acquired target area image is recorded; and / or, in response to a trigger operation of the recording control in the ninth interface, the audio data is recorded; and / or, in response to a trigger operation of the video recording control in the ninth interface, the video data corresponding to the target area image is recorded;

[0169] Correspondingly, the record sending module is also used to display a thumbnail of the target area image in the recorded editing state, a cover image of the audio data, and / or a cover image of the video data on the eighth interface.

[0170] In some optional implementations, the record sending module further includes:

[0171] A progress prompt submodule, used to display a sending progress control in a pop-up window interface during the process of sending a target area image corresponding to a thumbnail selected in an editing state, audio data corresponding to a cover image of audio data, and / or video data corresponding to a cover image of video data to an account of a target application;

[0172] The sending termination submodule is used to terminate the sending process in response to the triggering operation of canceling the sending control in the pop-up window interface;

[0173] The sending success prompt submodule is used to switch the pop-up window interface to the sending success prompt control when the sending is successful.

[0174] In some optional implementations, the second interface further displays at least one of the following controls: a learning content prompt control, a planned time prompt control, and an actual time prompt control;

[0175] Correspondingly, the application interaction device also includes:

[0176] The planned time-consuming editing module is used to edit the planned time-consuming in response to the triggering operation of the planned time-consuming prompt control in the second interface.

[0177] In some optional implementations, the application interaction device further includes:

[0178] The time-consuming display module is used to respond to the triggering operation of the completion control in the second interface, determine the actual time consumed to complete the learning content, and display the determined actual time consumed on the tenth interface.

[0179] The application interaction device provided in the embodiments of the present disclosure can execute the application interaction method provided in any embodiment of the present disclosure, and has the functional modules and beneficial effects corresponding to the execution method.

[0180] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present disclosure.

[0181] Embodiment 6

[0182] Reference below Fig.17 , which shows an electronic device (eg, Fig.17 The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.17 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0183] like Fig.17As shown, the electronic device 100 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 102 or a program loaded from a storage device 106 to a random access memory (RAM) 103. In the RAM 103, various programs and data required for the operation of the electronic device 100 are also stored. The processing device 101, the ROM 102, and the RAM 103 are connected to each other via a bus 104. An input / output (I / O) interface 105 is also connected to the bus 104.

[0184] Typically, the following devices may be connected to the I / O interface 105: an input device 106 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 107 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 108 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 109. The communication device 109 may allow the electronic device 100 to communicate with other devices wirelessly or by wire to exchange data. Fig.17 The electronic device 100 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0185] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 109, or installed from the storage device 106, or installed from the ROM 102. When the computer program is executed by the processing device 101, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0186] The electronic device provided by the embodiment of the present disclosure and the application interaction method provided by the above embodiment belong to the same public concept. The technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0187] Embodiment 7

[0188] An embodiment of the present disclosure provides a computer storage medium on which a computer program is stored. When the program is executed by a processor, the application interaction method provided by the above embodiment is implemented.

[0189] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM) or a flash memory (FLASH), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0190] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0191] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0192] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device:

[0193] In response to the triggering operation of the start control in the first interface, the target area image is collected, and the learning content of the target area image is recognized; the auxiliary tool control is matched based on the recognized learning content, and the matched auxiliary tool control is displayed on the second interface.

[0194] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), and / or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0195] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0196] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The names of the units and modules do not, in some cases, limit the units and modules themselves. For example, the tool matching module may also be described as an "auxiliary tool matching module".

[0197] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: Field Programmable Gate Array (FPGA), Application Specific Integrated Circuit (ASIC), Application Specific Standard Parts (ASSP), System on Chip (SOC), Complex Programmable Logic Device (CPLD), etc.

[0198] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0199] According to one or more embodiments of the present disclosure, [Example 1] provides an application interaction method, the method comprising:

[0200] In response to a triggering operation of a start control in the first interface, collecting a target area image, and performing learning content recognition on the target area image;

[0201] Auxiliary tool controls are matched based on the identified learning content, and the matched auxiliary tool controls are displayed on the second interface.

[0202] According to one or more embodiments of the present disclosure, [Example 2] provides an application interaction method, further comprising:

[0203] In some optional implementations, the performing learning content recognition on the target area image includes:

[0204] Performing text recognition on the target area image and determining the subject corresponding to the recognized text;

[0205] Correspondingly, matching the auxiliary tool controls based on the identified learning content includes: matching the auxiliary tool controls based on the determined disciplines.

[0206] According to one or more embodiments of the present disclosure, [Example 3] provides an application interaction method, further comprising:

[0207] In some optional implementations, the process of performing learning content recognition on the target area image further includes:

[0208] The target area image is displayed on a third interface, and a recognition animation is rendered on the displayed target area image.

[0209] According to one or more embodiments of the present disclosure, [Example 4] provides an application interaction method, further comprising:

[0210] In some optional implementations, the process of matching the auxiliary tool control based on the identified learning content further includes:

[0211] Recording the target area image according to a preset rule; and / or,

[0212] In response to a triggering operation of a photo control in the second interface, recording the currently acquired target area image; and / or,

[0213] In response to a triggering operation of a recording control in the second interface, recording audio data; and / or,

[0214] In response to a triggering operation of a video recording control in the second interface, recording video data corresponding to the target area image;

[0215] A record control is generated according to the record result, and the record control is displayed on the second interface.

[0216] According to one or more embodiments of the present disclosure, [Example 5] provides an application interaction method, further comprising:

[0217] In some optional implementations, recording the target area image according to a preset rule includes:

[0218] At preset time intervals, learning content recognition is performed on the currently acquired target area image, and / or clarity assessment is performed on the currently acquired target area image;

[0219] When the learning content is recognized and / or the clarity assessment meets the standard, the currently acquired target area image is recorded.

[0220] According to one or more embodiments of the present disclosure, [Example 6] provides an application interaction method, further comprising:

[0221] In some optional implementations, in response to a triggering operation of the recording control in the second interface, a thumbnail of the recorded target area image, a cover image of the audio data, and / or a cover image of the video data are displayed on the fourth interface.

[0222] According to one or more embodiments of the present disclosure, [Example 7] provides an application interaction method, further comprising:

[0223] In some optional implementations, if the fourth interface displays a thumbnail of the target area image, it also includes:

[0224] In response to the triggering operation of the thumbnail, displaying the target area image corresponding to the triggered thumbnail on the fifth interface;

[0225] In response to the viewing operation received on the fifth interface, a corresponding viewing action is performed on the currently displayed target area image.

[0226] According to one or more embodiments of the present disclosure, [Example 8] provides an application interaction method, further comprising:

[0227] In some optional implementations, if the fourth interface displays a cover image of the audio data, it also includes:

[0228] In response to a triggering operation of the cover image of the audio data, playing the audio data corresponding to the triggered cover image, and displaying relevant information of the played audio data on a sixth interface;

[0229] In response to the playback parameter control operation received by the sixth interface, playback parameter control is performed on the currently playing audio data.

[0230] According to one or more embodiments of the present disclosure, [Example 9] provides an application interaction method, further comprising:

[0231] In some optional implementations, if the fourth interface displays a cover image of the video data, it also includes:

[0232] In response to a triggering operation of the cover image of the video data, playing the video data corresponding to the triggered cover image in the seventh interface;

[0233] In response to the playback parameter control operation received on the seventh interface, playback parameter control is performed on the currently playing video data.

[0234] According to one or more embodiments of the present disclosure, [Example 10] provides an application interaction method, further comprising:

[0235] In some optional implementations, in response to completing the triggering operation of the control in the second interface, determining whether the target area image, audio data and / or video data is currently recorded;

[0236] If yes, then displaying a thumbnail of the target area image in the editing state, a cover image of the audio data, and / or a cover image of the video data on the eighth interface;

[0237] In response to the triggering operation of the send control in the eighth interface, the target area image corresponding to the thumbnail selected in the editing state, the audio data corresponding to the cover image of the audio data, and / or the video data corresponding to the cover image of the video data are sent to the account of the target application.

[0238] According to one or more embodiments of the present disclosure, [Example 11] provides an application interaction method, further comprising:

[0239] In some optional implementations, if no target area image, audio data and / or video data is currently recorded, a photo control, a sound recording control and / or a video recording control are displayed on the ninth interface;

[0240] In response to the triggering operation of the photo control in the ninth interface, recording the currently acquired target area image; and / or,

[0241] In response to a triggering operation of the recording control in the ninth interface, recording the audio data; and / or,

[0242] In response to the triggering operation of the video control in the ninth interface, the video data corresponding to the target area image is recorded;

[0243] A thumbnail of the target area image in the recorded editing state, a cover image of the audio data, and / or a cover image of the video data are displayed on the eighth interface.

[0244] According to one or more embodiments of the present disclosure, [Example 12] provides an application interaction method, further comprising:

[0245] In some optional implementations, in the process of sending the target area image corresponding to the thumbnail selected in the editing state, the audio data corresponding to the cover image of the audio data, and / or the video data corresponding to the cover image of the video data to the account of the target application, the process further includes:

[0246] Display the sending progress control in the pop-up window interface;

[0247] In response to a trigger operation of canceling the sending control in the pop-up window interface, terminating the sending process;

[0248] When the sending is successful, the pop-up window interface is switched to a sending success prompt control.

[0249] According to one or more embodiments of the present disclosure, [Example 13] provides an application interaction method, further comprising:

[0250] In some optional implementations, the second interface further displays at least one of the following controls: a learning content prompt control, a planned time prompt control, and an actual time prompt control;

[0251] Correspondingly, the method further includes: editing the planned time in response to a triggering operation of a planned time prompt control in the second interface.

[0252] According to one or more embodiments of the present disclosure, [Example 14] provides an application interaction method, further comprising:

[0253] In some optional implementations, in response to a triggering operation of completing a control in the second interface, the actual time consumed to complete the learning content is determined, and the determined actual time consumed is displayed on a tenth interface.

[0254] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0255] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0256] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. An application interaction method, characterized in that: include: In response to a triggering operation of a start control in the first interface, collecting a target area image, and performing learning content recognition on the target area image; Matching an auxiliary tool control based on the identified learning content, and displaying the matched auxiliary tool control on the second interface; The performing learning content recognition on the target area image includes: Performing text recognition on the target area image and determining the subject corresponding to the recognized text; Determining the subject corresponding to the recognized text includes: Matching the recognized text with text data corresponding to the learning content in a preset database, and determining the subject based on the matching result; The process of matching the auxiliary tool control based on the identified learning content also includes: Recording the target area image according to preset rules; The step of recording the target area image according to a preset rule includes: At preset time intervals, learning content recognition is performed on the currently acquired target area image, and / or clarity assessment is performed on the currently acquired target area image; When the identified learning content and / or the clarity assessment meets the standard, the currently acquired target area image is recorded; The second interface also displays an actual time-consuming prompt control, wherein the actual time-consuming prompt control is used to prompt the user of the actual time consumed for the current subject, and to reset the timing when a new subject is identified.

2. The method according to claim 1, characterized in that The matching of auxiliary tool controls based on the identified learning content includes: matching auxiliary tool controls based on the determined disciplines.

3. The method according to claim 1, characterized in that The process of performing learning content recognition on the target area image also includes: The target area image is displayed on a third interface, and a recognition animation is rendered on the displayed target area image.

4. The method according to claim 1, characterized in that: The process of matching the auxiliary tool control based on the identified learning content also includes: In response to a triggering operation of a photo control in the second interface, recording the currently acquired target area image; and / or, In response to a triggering operation of a recording control in the second interface, recording audio data; and / or, In response to a triggering operation of a video recording control in the second interface, recording video data corresponding to the target area image; A record control is generated according to the record result, and the record control is displayed on the second interface.

5. The method according to claim 4, characterized in that Also includes: In response to the triggering operation of the recording control in the second interface, a thumbnail of the recorded target area image, a cover image of the audio data, and / or a cover image of the video data are displayed on the fourth interface.

6. The method according to claim 5, characterized in that If the fourth interface displays a thumbnail of the target area image, the method further includes: In response to the triggering operation of the thumbnail, displaying the target area image corresponding to the triggered thumbnail on the fifth interface; In response to the viewing operation received on the fifth interface, a corresponding viewing action is performed on the currently displayed target area image.

7. The method according to claim 5, characterized in that If the fourth interface displays a cover image of audio data and / or a cover image of video data, it also includes: In response to a triggering operation of the cover image of the audio data, playing the audio data corresponding to the triggered cover image, and displaying relevant information of the played audio data on a sixth interface; In response to the playback parameter control operation received by the sixth interface, playback parameter control is performed on the currently playing audio data.

8. The method according to claim 5, characterized in that If the fourth interface displays a cover image of the video data, it also includes: In response to a triggering operation of the cover image of the video data, playing the video data corresponding to the triggered cover image in the seventh interface; In response to the playback parameter control operation received on the seventh interface, playback parameter control is performed on the currently playing video data.

9. The method according to claim 4, characterized in that Also includes: In response to completing the triggering operation of the control in the second interface, determining whether a target area image, audio data and / or video data is currently recorded; If yes, then displaying a thumbnail of the target area image in the editing state, a cover image of the audio data, and / or a cover image of the video data on the eighth interface; In response to the triggering operation of the send control in the eighth interface, the target area image corresponding to the thumbnail selected in the editing state, the audio data corresponding to the cover image of the audio data, and / or the video data corresponding to the cover image of the video data are sent to the account of the target application.

10. The method according to claim 9, characterized in that Also includes: If no target area image, audio data and / or video data is currently recorded, a photo taking control, a sound recording control and / or a video recording control are displayed on the ninth interface; In response to the triggering operation of the photo control in the ninth interface, recording the currently acquired target area image; and / or, In response to a triggering operation of the recording control in the ninth interface, recording the audio data; and / or, In response to the triggering operation of the video control in the ninth interface, the video data corresponding to the target area image is recorded; A thumbnail of the target area image in the recorded editing state, a cover image of the audio data, and / or a cover image of the video data are displayed on the eighth interface.

11. The method according to claim 9, characterized in that In the process of sending the target area image corresponding to the thumbnail selected in the editing state, the audio data corresponding to the cover image of the audio data, and / or the video data corresponding to the cover image of the video data to the account of the target application, the method further includes: Display the sending progress control in the pop-up window interface; In response to a trigger operation of canceling the sending control in the pop-up window interface, terminating the sending process; When the sending is successful, the pop-up window interface is switched to a sending success prompt control.

12. The method according to any one of claims 1 to 11, characterized in that: The second interface also displays at least one of the following controls: a learning content prompt control, a planned time prompt control; Correspondingly, the method further includes: editing the planned time in response to a triggering operation of a planned time prompt control in the second interface.

13. The method according to claim 12, characterized in that Also includes: In response to the triggering operation of completing the control in the second interface, the actual time consumed to complete the learning content is determined, and the determined actual time consumed is displayed on the tenth interface.

14. An application interaction device, characterized in that: include: An image recognition module, for collecting a target area image in response to a triggering operation of a start control in the first interface, and performing learning content recognition on the target area image; A tool matching module, used to match auxiliary tool controls based on the identified learning content, and display the matched auxiliary tool controls on the second interface; The image recognition module is specifically used to perform text recognition on the target area image and determine the subject corresponding to the recognized text; Determining the subject corresponding to the recognized text includes: Matching the recognized text with text data corresponding to the learning content in a preset database, and determining the subject based on the matching result; The application interaction device also includes: A recording module, used to record the target area image according to a preset rule in the process of matching the auxiliary tool control based on the identified learning content; The recording module comprises: The automatic recording submodule is used to identify the learning content of the currently acquired target area image and / or perform clarity assessment on the currently acquired target area image at preset time intervals; when the identified learning content and / or clarity assessment meet the standards, the currently acquired target area image is recorded; The second interface also displays an actual time-consuming prompt control, wherein the actual time-consuming prompt control is used to prompt the user of the actual time consumed for the current subject, and to reset the timing when a new subject is identified.

15. An electronic device, characterized in that: The electronic device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the application interaction method as described in any one of claims 1-13.

16. A storage medium comprising computer executable instructions, wherein the computer executable instructions are used to perform the application interaction method according to any one of claims 1 to 13 when executed by a computer processor.

Images