Positioning method, positioning device, electronic equipment and computer readable storage medium

Abstract

The invention relates to a positioning method, a positioning device, electronic equipment and a computer readable storage medium. The positioning method comprises the following steps: acquiring an image of a current interface; determining the position and the category of a specified element in the image; according to the component to be displayed in the current interface and the position and the category of the specified element, a display area of the component in the current interface is determined, and the display area and the area, in the image, of the specified element are not overlapped.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technologies, and more particularly, to a positioning method, a positioning device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the development of artificial intelligence (AI) technology and large language models, many software programs that can interact with AI-driven agents have emerged. Generally, in the development of a graphical user interface (GUI), software can provide users with interactive components to assist in the operation and use of the current interface. On the one hand, components with different functions need to be displayed in different functional scenarios. On the other hand, different software programs may display their respective components simultaneously. Therefore, how to position the display positions of the components will affect the user experience. Summary of the Invention

[0003] According to some embodiments of the present disclosure, there is provided a positioning method, including: obtaining an image of a current interface; determining the position and category of a specified element in the image; and determining a display area of a component in the current interface according to the component to be displayed in the current interface and the position and category of the specified element, wherein the display area does not overlap with the area of the specified element in the image.

[0004] According to some other embodiments of the present disclosure, there is provided a positioning device, including: an obtaining module configured to obtain an image of a current interface; a determining module configured to determine the position and category of a specified element in the image; and a display module configured to determine a display area of a component in the current interface according to the component to be displayed in the current interface and the position and category of the specified element, wherein the display area does not overlap with the area of the specified element in the image.

[0005] According to some embodiments of the present disclosure, there is provided an electronic device, including: a memory; and a processor coupled to the memory, the processor being configured to execute the method according to any one of the embodiments of the present disclosure based on instructions stored in the memory.

[0006] According to some embodiments of the present disclosure, there is provided a computer-readable storage medium having stored thereon a computer program, which when executed by a processor, executes the method according to any one of the embodiments of the present disclosure.

[0007] Other features, aspects, and advantages of the present disclosure will become apparent from the following detailed description of the exemplary embodiments of the present disclosure with reference to the accompanying drawings. Brief Description of the Drawings

[0008] Embodiments of the present disclosure will be described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in the following description only relate to some embodiments of the present disclosure and do not constitute a limitation on the present disclosure. In the drawings:

[0009] Figure 1 A flowchart showing a method for positioning components displayed in an interface according to some embodiments of the present disclosure;

[0010] Figure 2 A schematic diagram of an interface according to some embodiments of the present disclosure;

[0011] Figure 3 A schematic diagram showing the position of components displayed in an interface according to some embodiments of the present disclosure;

[0012] Figure 4 A schematic diagram showing the position of components displayed in an interface according to some other embodiments of the present disclosure;

[0013] Figure 5 A schematic diagram showing the position of components displayed in an interface according to still some other embodiments of the present disclosure;

[0014] Figure 6 A schematic block diagram showing a device for positioning components displayed in an interface according to some embodiments of the present disclosure;

[0015] Figure 7 A block diagram showing an electronic device according to some embodiments of the present disclosure;

[0016] Figure 8 A block diagram showing an electronic device according to some other embodiments of the present disclosure.

[0017] It should be understood that for the sake of description, the dimensions of the various parts shown in the drawings are not necessarily drawn in actual proportional relationship. The same or similar reference numerals are used in the various drawings to represent the same or similar components. Therefore, once an item is defined in one drawing, it may not be further discussed in the subsequent drawings. Detailed Embodiments

[0018] The technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present disclosure. It should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein.

[0019] It should be understood that the various steps described in the method embodiments of the present disclosure may be executed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this regard. Unless otherwise specifically stated, the relative arrangements of components and steps, numerical expressions and values set forth in these embodiments should be construed as merely exemplary and not limiting the scope of the present disclosure.

[0020] The term "comprising" and variations thereof used in the present disclosure mean open terms that include at least the subsequent elements / features, but do not exclude other elements / features, i.e., "including but not limited to". The term "based on" means "at least partially based on".

[0021] It should be noted that concepts such as "first", "second", etc. mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to define the order of functions performed by these devices, modules or units or their interdependencies. Unless otherwise specified, concepts such as "first", "second", etc. are not intended to imply that the objects so described must be in a given order in terms of time, space, ranking or any other way.

[0022] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0023] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only for illustrative purposes and do not limit the scope of these messages or information.

[0024] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present disclosure are all information and data authorized by the user or fully authorized by all parties. And the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of relevant countries and regions, and corresponding operation entrances are provided for users to choose to authorize or refuse.

[0025] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings, but the present disclosure is not limited to these specific embodiments. These specific embodiments may be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. In addition, in one or more embodiments, specific features, structures or characteristics may be combined in any suitable manner that will be apparent to those of ordinary skill in the art from the present disclosure.

[0026] It should be understood that the present disclosure also places no restrictions on how to obtain the image to be applied / processed. In some embodiments of the present disclosure, it can be obtained from a storage device, such as an internal memory or an external storage device. In some other embodiments of the present disclosure, a photographing component can be activated to take a picture. It should be noted that the obtained image can be a single captured image or a frame of a captured video, and is not particularly limited thereto.

[0027] In the context of the present disclosure, the image can refer to any one of a variety of images, such as a color image, a grayscale image, etc. It should be noted that in the context of this specification, the type of the image is not specifically limited. In addition, the image can be any suitable image, such as an original image obtained by an imaging device, or an image that has been subjected to specific processing on the original image, such as preliminary filtering, anti-aliasing, color adjustment, contrast adjustment, normalization, etc. It should be noted that the preprocessing operation can also include other types of preprocessing operations known in the art, which will not be described in detail here.

[0028] In the related art, software usually provides components with specific functions in the user's current interface, including but not limited to input control components for implementing user instruction capture (such as buttons, text input boxes, drop-down selectors, etc.), navigation components for managing interface jump logic (such as hierarchical navigation, responsive tab bars, collapsible tree menus, etc.), or feedback components for transmitting operation status (such as loading progress bars, operation success / error prompt boxes, etc.). These components guide and assist users in different scenarios, but they need to be positioned within the interface to avoid conflicts with other components and not affect the user's use by blocking important information in the current interface. For this reason, the present disclosure proposes a positioning method, in which other components and key information can be avoided in the current interface to assist users for reference and improve the user's use experience.

[0029] Specifically, Figure 1 FIG. shows a flowchart of a method for positioning a component displayed in an interactive interface according to some embodiments of the present disclosure.

[0030] As Figure 1 shown, in step S101, an image of the current interface is obtained; in step S102, the position and category of a specified element in the image are determined; in step S103, according to the component to be displayed in the current interface and the position and category of the specified element, a display area of the component in the current interface is determined, wherein the display area does not overlap with the area of the specified element in the image.

[0031] The recommended method of this embodiment can be executed on the client side or partially on the server side.

[0032] Generally, the current interface in the present disclosure is not limited to the interaction interface between the user and the agent. As long as it is various functional interfaces that the user normally uses, such as browser interfaces, office software interfaces, social software interfaces, etc., the corresponding functional components may be called. The designated elements are mainly used to describe the key content in the current interface, including the title of the interface, the menu, or other highlighted text and multimedia resources, etc., and may also include various categories such as components of other software. The positions of these designated elements in the interface may conventionally be at the top of the interface, or at specific positions that are dynamically changing according to the scenario or requirements of the interface. The display area is mainly used to describe the display position of the component in the interface, and additionally, the appearance characteristics of the component need to be considered, such as the size, color matching, etc. of the component.

[0033] The positioning method provided by the present disclosure is generally applied to the components that need to be displayed in the interface by the software. Specifically, reference can be made to Figure 2 , which shows a schematic diagram of the interface 1 currently to be operated by the user according to some embodiments of the present disclosure. Generally, the interface 1 includes at least one of a function module 12, a title module 13, and a content module 14. The function module 12 is mainly used to operate on the interface content, such as the function areas where the editing function, insertion function, layout function, review function, etc. of the office software interface are located, and the setting areas where the address navigation, web page collection, web page editing, etc. of the browser interface are located. The function module 12 is usually located at the top of the interface, and may also appear in the form of a sidebar on both sides of the interface. The title module 13 mainly includes interface content such as titles, usually text with specific font, specific font size, etc. in the interface, and may also be multimedia resources with a leading or generalizing nature, such as images or videos as "push head pictures", etc. The title module 13 is generally located below the function module 12 and before the content module 14, and gradually moves as the user scrolls the interface, and may also be located at a fixed position such as the bottom of the interface. The content module 14 usually includes the content text of the interface 1, and may also include multimedia resources such as static pictures, dynamic images, short videos, slides, virtual reality / augmented reality content, social media content, animated films, interactive elements, etc. It can be understood that the interface 1 does not necessarily include the above three modules at the same time, and the positions of the modules will change during the user's use of the interface 1, and may switch between display and non-display according to the user's input operations; additionally or alternatively, in the case of different scenario functions, the image of the interface 1 may also include other types of modules.

[0034] Further, in some embodiments, determining the display area of the component in the current interface includes: using a machine learning model to perform image recognition on the image to obtain first key information in the image; and determining the display area based on the first key information, where the display area does not overlap with the first key information. As Figure 2As shown, interface 1 includes component 15 displayed near title module 13 and content module 14. Component 15 is an input control component for the interface content, mainly including functions such as "Search", "Read Aloud", and "Others". First, obtain the image of the current interface 1. For example, methods similar to screen capture can be used. At this time, the specified elements in the image may include title module 13 and / or content module 14, located in the middle and the lower middle of interface 1 respectively. Determine the display area of component 15 in the current interface 1 by performing image recognition on the image using a machine learning model. Then, the first key information in the image can be obtained as the main content of the interface displayed by title module 14. Based on this, the display area of component 15 needs to not overlap with content module 14 where the first key information is located, so as to determine the position of component 15. It can be understood that the first key information here is only an example. In fact, the machine learning model can be trained according to requirements to adjust the output type of image recognition, and different key parts of the interface can be obtained as the first key information.

[0035] Additionally or alternatively, the category of the specified element may affect the position of the component to be determined: based on the category of the specified element, determine the relative position relationship between the component and the specified element in the current interface; and based on the position and relative position relationship of the specified element, determine the position of the display area. Specifically, in response to the category of the specified element being an avoidance element, determine that the display area is far from the specified element in the current interface; or in response to the category of the specified element being an associated element, determine that the display area is close to the specified element in the current interface. In a non-limiting embodiment, refer to Figure 2 , when the specified element is function module 12, title module 13, or content module 14 in interface 1, it belongs to the associated element corresponding to the component. Then, the component to be displayed should be close to the corresponding specified element on the basis of non-overlap, so that the user can use the specified element and the component in an associated manner. Conversely, in another non-limiting embodiment, continue to refer to Figure 2 , when the specified element is component 15 in interface 1, it belongs to an avoidance element that may conflict with another component (not shown). Then, when determining the display position of another component, it is necessary to not only ensure that another component does not overlap with component 15, but also preferably be far from component 15, so as not to show a crowded layout in the interface, affecting the user's visual experience and usage experience.

[0036] In some embodiments, users usually have certain inputs for the current interface, including but not limited to editing and browsing the interface content, querying and using the interface functions, etc. The category of the input may affect the category of the component to be displayed on the interface and / or the category of the specified element for which image recognition needs to be performed; the position of the input may affect the position of the component, etc.

[0037] Specifically, in some embodiments, the user's input can be used as a trigger for obtaining an interface image, that is, in response to the user's input in the current interface, an image of the current interface is obtained. The user may have different input operations for interfaces in different functional scenarios. The position, content, and action of the input will affect the components to be displayed. That is, the components will be displayed only after the input. Therefore, obtaining the interface image after the input helps to determine the display area of the components relatively in real time. Next, please refer to Figures 3 to 5 , which respectively show schematic diagrams of the positions of components displayed in an interaction interface according to some embodiments of the present disclosure.

[0038] In some embodiments, determining the position and category of a specified element in an image includes: using a machine learning model to perform image recognition on a predetermined range around the input position in the image to determine the position of the specified element. The input includes selecting the content at the input position in the current interface. The category of the component is the same as that of the specified element, and the source of the component is different from that of the specified element.

[0039] As Figure 3 shown, interface 1 mainly includes content module 14. At this time, the user's input includes selecting the content at the input position in the current interface. Generally, when the user selects text in the interface content, it is for copying or querying, etc. The interactive software will display relevant components near the selected text 16 according to the position of the user's input to facilitate the user's subsequent operations and improve the operation efficiency. For the component 17 to be displayed, other software may also provide components 15 with similar functions. In other words, from a functional perspective, the categories of component 17 and component 15 are the same, and because they belong to two different interactive software, their sources are different. At this time, in the image of interface 1, the specified element is component 15, which belongs to an avoidance element. On the basis of non-overlap, it is determined that component 17 is located on the opposite side of the selected text 16 from component 15. Component 17 can provide appropriate function assistance to the user while avoiding display conflicts with other components, so as not to affect the user's visual experience and usage experience.

[0040] Based on this, the positioning logic of the specified element (component 15) in the display position is actually the same as that of the component 17 to be displayed, that is, it also needs to appear near the selected text. Therefore, image recognition can be performed on the predetermined range 11 around the user input position to determine the position of the specified element. The predetermined range 11 can be reduced from the full-screen image of interface 1 to the area near the input position, thereby significantly reducing the computational amount of image recognition and improving the speed and efficiency of determining the specified element. Specifically, the position and size of the predetermined range 11 can be centered on the position of the selected text 16 and expanded by a specified number of pixel points around it. It should be understood that the number of pixel points in the predetermined range 11 is mainly determined according to the user's usage habits and the interface design style.

[0041] In some embodiments, determining the position and category of a specified element in an image includes: determining the category of the specified element based on the input. Additionally, determining the display area of a component in the current interface includes: determining the category of the component to be displayed in the current interface based on the input, and determining the display area of the component in the current interface based on the category of the component. As previously mentioned, as Figure 3 shown, in response to the input including selecting the content at the input position, it is determined that the component is an input control component, and the category of the specified element is another component with the same category as the component but different sources, so as to determine that the position of the component is on the other side of the selected text different from other components.

[0042] Alternatively, in response to the input content including querying a specified function in the current interface, it is determined that the category of the specified element is the operation menu of the specified function, and the category of the component is the query dialog box. As Figure 4 shown, interface 1 mainly includes function module 12, which includes the operation menu of specified function 121; the user's input includes querying specified function 121 in the current interface 1.

[0043] In a non-limiting embodiment, when using, for example, interactive software between the user and the agent for function query, the user will input the content of "how to use the specified function" in component 18 provided by the interactive software. In the image of the current interface 1, the specified element is the operation menu of specified function 121, and the position is at the top of interface 1; component 18 is the query dialog box, and the initial position (not shown) can be provided at any position in interface 1 according to the user's usage habits, such as the center or side of page 1. After receiving the user's input, on the one hand, descriptive text on how to use specified function 121 is provided in component 18, and on the other hand, on the basis of ensuring that component 18 does not overlap with specified function 121, it is determined that the display area of component 18 in interface 1 is near specified function 121. It can be understood that the descriptive text in component 18 is only an example, and component 18 may have various forms of responses to the function query input by the user, including but not limited to text about function introduction, multimedia resources, or combinations thereof, etc. The display position of component 18 is beneficial to provide directional prompts for the user, facilitating the user to quickly locate the queried function in the interface and referring to component 18 for subsequent operations, thereby enhancing the user's usage experience of the component.

[0044] Alternatively, in response to the input operation including browsing the current interface, it is determined that the category of the specified element is the first key information of the current interface, and the category of the component is the browsing auxiliary menu. Figure 5As shown, interface 1 mainly includes a title module 13 and a content section 14. The user's input includes browsing in the current interface 1. The input actions may include keeping the mouse scrolling for a certain period of time and / or keeping the mouse hovering for a certain period of time after the mouse scrolls. These actions can reflect that the user is browsing the page content or carefully reading some content of the current interface.

[0045] In a non-limiting embodiment, when the user hovers the mouse as described above while reading the content of the current interface 1, in the image of interface 1 obtained at this time, the specified element is the title in the title module 13 that serves as the second key information, and the position is at the top of interface 1; the component 19 is a browsing assistance menu, including a text prompt "Do you need to open AI reading for you?" and a button "Use AI reading". The operation of obtaining the image can be based on the judgment of the mouse hovering duration. If the duration meets a predetermined threshold, the image of the current interface 1 is obtained. Image recognition is performed on the image to obtain the position of the title module 13. On the basis of ensuring that the component 19 does not overlap with the title module 13, the display area of the component 19 in the interface 1 is determined to be near the title module 13. It can be understood that similar to the first key information, the type of the second key information can also include various types. Figure 6 The title module 13 in [description] is only an example. Actually, other text or multimedia resources in the interface 1 can be determined as the second key information according to requirements to determine the position and category of the specified element and the component 19.

[0046] In some embodiments, it is not necessary to use the user's input as a trigger method for obtaining the image of the interface. The specified element can be directly determined based on the image of the interface 1. At this time, determining the position and category of the specified element in the image includes: using a machine learning model to perform image recognition on the image to obtain the second key information; and based on the second key information, determining the position and category of the specified element. Further, determining the display area of the component in the current interface includes: based on the category of the specified element, determining the category of the component; and based on the category of the component, determining the display area of the component in the current interface. Among them, in response to the specified element including the second key information of the current interface, the component is determined to be a browsing assistance menu. Still referring to Figure 5 , without user input, directly obtain the image of the interface 1. Use a machine learning model to perform image recognition on the image to obtain the title as the second key information, so as to similarly determine that the category of the component 19 is a browsing assistance menu, provide the user with an option to turn on AI reading aloud, and determine the display position of the component 19 near the title module 13. It can be understood that the interface image can be obtained at any time point after the current interface is loaded, such as immediately obtaining the image after the interface content is loaded, or the interface image can be obtained after setting a predetermined waiting period. Specifically, it can be determined according to the user's usage habits or interface design.

[0047] Further, please refer to Figure 6 , which shows a schematic block diagram of a positioning device for displaying components in an interface according to some embodiments of the present disclosure. Specifically, the positioning method can be implemented by the positioning device 6. The positioning device 6 includes a processor and a memory (not shown), where the processor can refer to various implementations of digital circuit systems, analog circuit systems, or mixed-signal (a combination of analog and digital) circuit systems that perform functions in a computing system. The processing circuit can include, for example, circuits such as integrated circuits (ICs), application-specific integrated circuits (ASICs), parts or circuits of a single processor core, the entire processor core, a single processor, programmable hardware devices such as field-programmable gate arrays (FPGAs), and / or systems including multiple processors. Additionally, the memory of the recommendation device 4 can store information generated by the processor and programs and data for the operation of the processor. The memory can be volatile memory and / or non-volatile memory. For example, the memory can include, but is not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), read-only memory (ROM), and flash memory. Generally, the processor can be configured to execute instructions stored in the memory to implement the recommendation method for user-agent interaction in the present disclosure.

[0048] Specifically, as Figure 6 shown, in some embodiments, the positioning device 6 of the present disclosure can include an acquisition module 61, a determination module 62, and a display module 63. Specifically, the acquisition module 61 is configured to acquire an image of the current interface; the determination module 62 is configured to determine the position and category of a specified element in the image; and the display module 63 is configured to determine the display area of the component in the current interface according to the component to be displayed in the current interface, as well as the position and category of the specified element, where the display area does not overlap with the area of the specified element in the image.

[0049] The present disclosure also provides a positioning device, which can include a processor, and a processor coupled to a memory, and the processor is configured to execute the positioning method for displaying components in an interactive interface according to any one of the foregoing embodiments of the present disclosure based on instructions stored in the memory. The positioning device can refer to Figure 7 , which shows a block diagram of an electronic device according to some embodiments of the present disclosure.

[0050] As Figure 7As shown, the memory 71 is used to store one or more computer-readable instructions. The memory 71 may include any combination of various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory, including but not limited to random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), read-only memory (ROM), and flash memory. For example, the memory 71 may store an operating system, application programs, a boot loader, a database, and other programs, and may also store various application programs and various data, etc.

[0051] The processor 72 is used to run computer-readable instructions to implement the song screening method described in any of the foregoing embodiments or the method described in any of the foregoing embodiments. For the specific implementation of each step of the method, reference may be made to the foregoing embodiments, and the repeated parts will not be elaborated herein.

[0052] The processor 72 may be configured to execute Figures 1 to 6 each step in. The processor 72 may be embodied as various processing devices, such as a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The central processing unit (CPU) may be of the X86 or ARM architecture, etc.

[0053] The processor 72 and the memory 71 may communicate with each other directly or indirectly. For example, the processor 72 and the memory 71 may communicate through a network. The network may include a wireless network, a wired network, and / or any combination of a wireless network and a wired network. The processor 72 and the memory 71 may also communicate with each other through a system bus, and the present disclosure does not limit this.

[0054] It should be noted that Figure 7 the components of the electronic device 7 shown are exemplary and not restrictive. According to actual application needs, the electronic device 7 may also have other components. The processor 72 may control other components in the electronic device 7 to perform desired functions.

[0055] The electronic device 7 may be implemented in a software, firmware, and / or hardware manner and may be integrated in a device installed with relevant application programs.

[0056] Figure 8 A block diagram of an electronic device according to other embodiments of the present disclosure is shown.

[0057] Figure 8The electronic device 8 shown can be a computer system with a dedicated hardware structure, which can perform corresponding functions when relevant application programs are installed.

[0058] The electronic device includes, but is not limited to, mobile terminals such as smart phones, laptop computers, personal digital assistants (PDAs), tablet personal computers (Tablet PCs), portable multimedia players (PMPs), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), wearable devices, etc., and fixed terminals such as digital TVs, desktop computers, etc.

[0059] As Figure 8 shown, the central processing unit (CPU) 81 executes various processes according to the programs stored in the read-only memory (ROM) 82 or the programs loaded from the storage section 88 into the random access memory (RAM) 83. In the RAM 83, data required when the CPU 81 executes various processes, etc., is stored as needed. The central processing unit is merely exemplary, and it can also be other types of processors, such as the various processors described above. The ROM 82, RAM 83, and storage section 88 can be various forms of computer-readable storage media. It should be noted that although Figure 8 the ROM 82, RAM 83, and storage section 88 are shown separately, one or more of them can be combined or located in the same or different memories or storage modules.

[0060] The CPU 81, ROM 82, and RAM 83 are connected to each other via the bus 84. The input / output interface 85 is also connected to the bus 84.

[0061] The following components are connected to the input / output interface 85: an input section 86, such as a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; an output section 87, including a display, such as a cathode ray tube (CRT), liquid crystal display (LCD), speaker, vibrator, etc.; a storage section 88, including a hard disk, magnetic tape, etc.; and a communication section 89, including a network interface card such as a LAN card, modem, etc. The communication section 89 allows communication processing to be performed via a network such as the Internet. It is easy to understand that although Figure 8 some parts in the electronic device 8 are shown to communicate via the bus 84, they can also communicate via a network or other means, where the network can include a wireless network, a wired network, and / or any combination of a wireless network and a wired network.

[0062] As needed, the drive 810 is also connected to the input / output interface 85. A removable medium 811 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is installed on the drive 810 as needed, so that the computer program read therefrom is installed in the storage section 88 as needed.

[0063] In the case where the above-described series of processes are implemented by software, the program constituting the software can be installed from a network such as the Internet or a storage medium such as the removable medium 811.

[0064] According to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product that, when run on a computer, causes the computer to implement the method described in any of the foregoing embodiments. The computer program product includes computer instructions carried on a computer-readable medium, including program code for executing the method shown in the flowchart. In such an embodiment, the computer instructions can be downloaded and installed from a network through the communication section 89, or installed from the storage section 88, or installed from the ROM 82. When the computer program is executed by the CPU 81, the method of the embodiment of the present disclosure is executed.

[0065] It should be noted that, in the context of the present disclosure, a computer-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0066] The computer-readable medium can be a computer-readable storage medium, a computer-readable signal medium, or any combination of the two.

[0067] The computer-readable storage medium includes, but is not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device. Computer instructions are stored on the computer-readable storage medium, and when the instructions are executed by a processor, the method described in any of the foregoing embodiments is implemented.

[0068] A computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, and the computer-readable signal medium may send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.

[0069] The foregoing computer-readable medium may be included in the aforementioned electronic device; or may exist separately without being assembled into the electronic device.

[0070] In some embodiments, a computer program is also provided, including: instructions that, when executed by a processor, cause the processor to execute the method described in any of the foregoing embodiments. For example, the instructions may be embodied as computer program code.

[0071] In embodiments of the present disclosure, computer program code for performing the operations of the present disclosure may be written in one or more programming languages or combinations thereof. The foregoing programming languages include but are not limited to object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., by using an Internet service provider to connect through the Internet).

[0072] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a portion of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks may occur in a different order than that noted in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0073] The functions described above can be performed, at least in part, by one or more hardware logic components. By way of example, and without limitation, exemplary hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0074] Although some specific embodiments of the present disclosure have been described in detail by way of example, those skilled in the art should understand that the above examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Those skilled in the art should understand that the above embodiments can be modified without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. A positioning method, comprising: Get the image of the current interface; determining a location and a category of a specified element in the image; as well as According to the component to be displayed in the current interface and the position and category of the designated element, a display area of ​​the component in the current interface is determined, wherein the display area does not overlap with an area of ​​the designated element in the image.

2. The positioning method according to claim 1, wherein: Determining a display area of ​​the component in the current interface includes: Using a machine learning model, performing image recognition on the image to obtain first key information in the image; and The display area is determined based on the first key information, wherein the display area does not overlap with the first key information.

3. The positioning method according to claim 1, wherein: Determining a display area of ​​the component in the current interface includes: Based on the category of the designated element, determining the relative position relationship between the component and the designated element in the current interface; and Based on the position of the designated element and the relative position relationship, the position of the display area is determined.

4. The positioning method according to claim 1, wherein: Determining a display area of ​​the component in the current interface includes: In response to the category of the designated element being an avoidance element, determining that the display area is away from the designated element in the current interface; or In response to the category of the designated element being an associated element, it is determined that the display area is close to the designated element in the current interface.

5. The positioning method according to any one of claims 1 to 4, wherein: Getting the image of the current interface includes: In response to user input in the current interface, an image of the current interface is acquired.

6. The positioning method according to claim 5, wherein: Determining the location and category of a specified element in the image includes: Using a machine learning model, image recognition is performed on a predetermined range around the input position in the image to determine the position of the designated element.

7. The positioning method according to claim 5, wherein: The input includes selecting content at the input position in the current interface, a category of the component is the same as a category of the designated element, and a source of the component is different from a source of the designated element.

8. The positioning method according to claim 5, wherein: Determining the location and category of a specified element in the image includes: Based on the input, a category of the specified element is determined.

9. The positioning method according to claim 8, wherein: Based on the input, determining the category of the specified element includes: In response to the input content including querying a designated function in the current interface, determining that the category of the designated element is an operation menu of the designated function.

10. The positioning method according to claim 8, wherein: Based on the input, determining the category of the specified element includes: The operation in response to the input includes browsing the current interface and determining the category of the designated element as the first key information of the current interface.

11. The positioning method according to claim 5, wherein: Determining a display area of ​​the component in the current interface includes: Based on the input, determining the category of the component to be displayed on the current interface; and Based on the category of the component, a display area of ​​the component in the current interface is determined.

12. The positioning method according to claim 11, wherein: Determining the category of the component to be displayed on the current interface includes: In response to the input comprising querying a specified function in the current interface, determining that the category of the component is a query dialog box; or In response to the input comprising browsing the current interface, determining the category of the component as a browsing auxiliary menu.

13. The positioning method according to any one of claims 1 to 4, wherein: Determining the location and category of a specified element in the image includes: Using a machine learning model, performing image recognition on the image to obtain second key information; and Based on the second key information, a position and a category of the designated element are determined.

14. The positioning method according to any one of claims 1 to 4, wherein: Determining a display area of ​​the component in the current interface includes: Determining a category of the component based on the category of the designated element; and Based on the category of the component, a display area of ​​the component in the current interface is determined.

15. The positioning method according to claim 14, wherein: Determining the category of the component includes: In response to the designated element including the second key information of the current interface, the component is determined to be a browsing auxiliary menu.

16. A positioning device, comprising: An acquisition module, configured to acquire an image of the current interface; A determination module, configured to determine a location and a category of a specified element in the image; as well as The display module is configured to determine a display area of ​​the component in the current interface according to the component to be displayed in the current interface and the position and category of the designated element, wherein the display area does not overlap with an area of ​​the designated element in the image.

17. An electronic device comprising: Memory; as well as A processor coupled to the memory, the processor being configured to execute the positioning method according to any one of claims 1 to 16 based on instructions stored in the memory.

18. A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the positioning method according to any one of claims 1 to 16 is performed.

19. A computer program product comprising computer executable instructions, which, when executed by a processor, cause the processor to implement the positioning method according to any one of claims 1 to 16.

Images