A key mapping method, electronic device and system

Abstract

This application provides a button mapping method, electronic device, and system. According to the method provided, the electronic device can identify button areas in a game application using an image processing algorithm. Furthermore, to improve the accuracy of button area recognition, the electronic device can also adjust the image processing algorithm based on the user's touch frequency on the game screen. After identifying all button areas, the user can match all virtual buttons in the game application with the physical buttons on the game controller, establishing a mapping relationship. The user can then trigger the functions corresponding to the virtual buttons in the game application by controlling the physical buttons on the game controller, such as character movement, skill activation, and scene switching. Implementing the method provided in this application simplifies the user's steps in matching game controller buttons with game application buttons, improving the efficiency and accuracy of button matching, and enhancing the user experience.

Description

Technical Field

[0001] This invention relates to the field of terminal technology, and in particular to a key mapping method, electronic device, and system. Background Technology

[0002] Most smartphones nowadays can connect to external gamepads, allowing users to control button-based applications (such as games) on their phones. In existing technology, pairing a phone with a gamepad involves first establishing a wireless connection via Bluetooth. Then, the phone screen displays the gamepad's button library interface, which includes multiple button icons. The user selects a button icon and manually drags it to the desired location on the phone screen. Once the button icon is hovered over the designated position, the user releases their finger, thus completing the pairing process.

[0003] As can be seen, in the aforementioned existing technology, users need to drag the button icon multiple times. Furthermore, users also need to drag the button icon to a designated location within the application. If the button icon is not hovered in an accurate position, the button mapping relationship between the phone and the peripheral controller may fail to be established. Therefore, this existing technology is cumbersome and inconvenient for users, with low button mapping matching efficiency and a poor user experience. Summary of the Invention

[0004] The purpose of this application is to provide a key mapping method, electronic device, and system that makes the process of establishing key mapping between a first electronic device and a second electronic device more intuitive, simple, and effective, greatly improving the efficiency of key mapping, simplifying user operation steps, and enhancing user experience.

[0005] The aforementioned and other objectives will be achieved through the features described in the independent claims. Further implementations are illustrated in the dependent claims, the specification, and the drawings.

[0006] In a first aspect, this application provides a method for key mapping, which may include: a first electronic device establishing a first connection with a second electronic device; the first electronic device displaying a first user interface, which may display multiple controls, including a first control; the first electronic device detecting a first user operation (e.g., a user tapping an icon with their finger); in response to the first user operation, the first electronic device recognizing the multiple controls in the first user interface; and the first electronic device selecting the first control among the recognized multiple controls. Then, in response to a first signal received when the first control is selected, the first electronic device establishes a mapping relationship between a first physical key and the first control.

[0007] In this application, the second electronic device may be a game controller. The game controller has one or more physical buttons and also includes a Bluetooth (BT) module and / or a wireless local area network (WLAN) module. The Bluetooth (BT) module can provide one or more Bluetooth communication solutions, including classic Bluetooth (Bluetooth 2.1 standard) or Bluetooth Low Energy (BLE). The WLAN module can provide one or more WLAN communication solutions, including wireless fidelity direct (Wi-Fi direct), wireless fidelity local area networks (Wi-Fi LAN), or wireless fidelity software access point (Wi-Fi softAP).

[0008] In this application, the first user interface can be the user interface of a game application, which is an application on electronic devices such as smartphones and tablets that provides entertainment for users. This application does not limit the name of the application. That is to say, the game application can be any game application available on the market that users can obtain and control, for example… However, this application does not impose any restrictions on this.

[0009] Implementing the first method enables the first electronic device to more efficiently and quickly identify controls in the first user interface and establish a mapping relationship with physical buttons on the second electronic device, thereby improving the efficiency of button mapping, enhancing the user experience, and making button mapping more accurate.

[0010] In conjunction with the first aspect, in some embodiments, the first connection may be a wireless connection established between the first electronic device and the second electronic device via one or more wireless communication technologies such as Bluetooth, Wi-Fi Direct, or Wi-Fi SoftAP, or a wired connection established between the first electronic device and the second electronic device via a universal serial bus (USB). After the first electronic device and the second electronic device establish a communication connection, the first electronic device may send data information to the second electronic device and / or receive data information from the second electronic device via one or more WLAN communication technologies such as USB, Bluetooth, Wi-Fi Direct, or Wi-Fi SoftAP.

[0011] In conjunction with the first aspect, in some embodiments, after establishing the mapping relationship between the first physical button and the first control of the second electronic device, the first electronic device can select a second control from a plurality of controls. Then, the first electronic device can receive a second signal sent by the second electronic device through a first connection, and in response to the second signal received when the second control is selected, establish the mapping relationship between the second physical button and the second control of the second electronic device. The second signal is generated by the second electronic device when the second physical button is pressed by the user.

[0012] In conjunction with the first aspect, in some embodiments, after the mapping relationship between multiple controls on the first electronic device and multiple physical buttons on the second electronic device is established, the first electronic device executes the corresponding function of the first control through the first signal received by the first connection.

[0013] In conjunction with the first aspect, in some embodiments, the first electronic device performs grayscale processing on the first user interface to obtain a first image. Then, the first electronic device can identify the control boundaries contained in the first user interface from the first image using an edge detection algorithm.

[0014] In conjunction with the first aspect, in some embodiments, the multiple controls in the first user interface may include a first control, which may include multiple boundary pixels, and the multiple boundary pixels may include the first pixel. The higher the frequency with which the touch position corresponding to the first pixel in the first user interface is touched by the user, the higher the probability that the first pixel is identified as the boundary pixel of the first control.

[0015] In conjunction with the first aspect, in some embodiments, when the first electronic device identifies the boundary of the first control in the first user interface from the first image using an edge detection method, the first electronic device may employ an edge operator to calculate the gradient vector of each pixel in the first image. Then, the first electronic device may employ linear interpolation to compare the gradient value of the first gradient vector of the first pixel with the gradient values ​​of other pixels in the same direction. If the gradient value of the first pixel is the largest in the stated direction, the first electronic device may retain the gradient value of the first pixel and set the gradient values ​​of the other pixels to zero. Next, the first electronic device may set a first threshold; if the gradient value of the first pixel is greater than the first threshold, the first pixel is retained, and all retained pixels constitute the boundary of the first control in the first user interface.

[0016] In conjunction with the first aspect, in some embodiments, when the first electronic device performs the function corresponding to the first control, the first electronic device displays a second user interface. This second user interface may be partially or entirely different from the aforementioned first user interface. Specifically, the second user interface may be a jump to a game application interface, the movement of a character within the game application interface, a change in the game scene, etc.

[0017] In conjunction with the first aspect, in some embodiments, when the first control in the first user interface is selected, the first control may be highlighted, or a cursor may be displayed in the area of ​​the first control, and / or the first control may be blinking. That is to say, this application does not limit how to prompt the user that the first control is selected.

[0018] In conjunction with the first aspect, in some embodiments, after the first electronic device displays a first user interface, a first floating control may be displayed on the first user interface. The first electronic device may detect a first user action (e.g., a click) performed on the first floating control.

[0019] Secondly, embodiments of this application provide a communication method applied to a communication system, which includes a first electronic device and a second electronic device. The first electronic device can establish a first connection with the second electronic device. The first electronic device can display a first user interface, which may include multiple controls, including a first control. The first electronic device can detect a first user operation, and in response to the first user operation, the first electronic device can identify the multiple controls in the first user interface. Then, the first electronic device can select the first control among the multiple controls, and the second electronic device can detect that a first physical button is pressed by the user, generate a first signal, and send the first signal to the first electronic device through the first connection. Next, the first electronic device can establish a mapping relationship between the first physical button and the first control in response to the first signal received when the first control is in a selected state.

[0020] Implementing the second method enables the first electronic device to more efficiently and quickly identify controls in the first user interface and establish a mapping relationship with physical buttons on the second electronic device, thereby improving the efficiency of button mapping, enhancing the user experience, and making button mapping more accurate.

[0021] In conjunction with the second aspect, in some embodiments, after establishing the mapping relationship between the first physical button and the first control, the first electronic device can select a second control from a plurality of controls. Then, the second electronic device can detect that the second physical button has been pressed by the user, generate a second signal, and send the second signal to the first electronic device through the first connection. Subsequently, the first electronic device can respond to the second signal received when the second control is in a selected state and establish the mapping relationship between the second physical button and the second control.

[0022] In conjunction with the second aspect, in some embodiments, the second electronic device in the communication system may be a game controller. A description of the game controller can be found in the description of the second electronic device provided in the first aspect above, and will not be repeated here.

[0023] In conjunction with the second aspect, in some embodiments, after the mapping relationship between multiple controls on the first electronic device and multiple physical buttons on the second electronic device is established, the first electronic device executes the corresponding function of the first control through the first signal received by the first connection.

[0024] In conjunction with the second aspect, in some embodiments, the first electronic device performs grayscale processing on the first user interface to obtain a first image. Then, the first electronic device can identify the control boundaries contained in the first user interface from the first image using an edge detection algorithm.

[0025] In conjunction with the second aspect, in some embodiments, the first electronic device performs grayscale processing on the first user interface to obtain a first image. Then, the first electronic device can identify the control boundaries contained in the first user interface from the first image using an edge detection algorithm.

[0026] In conjunction with the second aspect, in some embodiments, the multiple controls in the first user interface may include a first control, which may include multiple boundary pixels, and the multiple boundary pixels may include the first pixel. The higher the frequency with which the touch position corresponding to the first pixel in the first user interface is touched by the user, the higher the probability that the first pixel is identified as the boundary pixel of the first control.

[0027] In conjunction with the second aspect, in some embodiments, when the first electronic device identifies the boundary of the first control in the first user interface from the first image using an edge detection method, the first electronic device can employ an edge operator to calculate the gradient vector of each pixel in the first image. Then, the first electronic device can employ linear interpolation to compare the gradient value of the first gradient vector of the first pixel with the gradient values ​​of other pixels in the same direction. If the gradient value of the first pixel is the largest in that direction, the first electronic device can retain the gradient value of the first pixel and set the gradient values ​​of other pixels to zero. Next, the first electronic device can set a first threshold. If the gradient value of the first pixel is greater than the first threshold, the first pixel is retained, and all retained pixels constitute the boundary of the first control in the first user interface.

[0028] In conjunction with the second aspect, in some embodiments, when the first electronic device performs the function corresponding to the first control, the first electronic device displays a second user interface. This second user interface may be partially or entirely different from the aforementioned first user interface. Specifically, the second user interface may be a jump to a game application interface, the movement of a character within the game application interface, a change in the game scene, etc.

[0029] In conjunction with the second aspect, in some embodiments, when the first control in the first user interface is selected, the first control may be highlighted, a cursor may be displayed in the area of ​​the first control, and / or the first control may be blinking. That is to say, this application does not limit how the user is notified that the first control is selected. In conjunction with the second aspect, in some embodiments, after the first electronic device displays the first user interface, a first floating control may be displayed on the first user interface. The first electronic device can detect a first user operation (e.g., a click) applied to the first floating control.

[0030] Thirdly, embodiments of this application provide an electronic device, which may include: a communication device, a touch screen, a memory, and a processor coupled to the memory, wherein the memory stores computer-executable instructions. The communication device may be used to establish a first connection with a second electronic device. The touch screen may be used to display a first user interface, which may include multiple controls, including a first control. The touch screen may also be used to detect a first user operation. The processor may be used to identify the multiple controls in the first user interface and to select the first control among the multiple controls. The communication device may also be used to receive a first signal sent by the second electronic device through the first connection. The processor may also be used to, in response to the first signal received when the first control is selected, establish a mapping relationship between a first physical button and a first control in the second electronic device, wherein the first signal is generated by the second electronic device when the first physical button is pressed by a user.

[0031] In conjunction with the third aspect, in some embodiments, the processor can also be used to select a second control from a plurality of controls after establishing a mapping relationship between a first physical button and a first control of the second electronic device. The communication device can also be used to receive a second signal sent by the second electronic device via a first connection. The processor can also be used to establish a mapping relationship between a second physical button and a second control of the second electronic device in response to the second signal received when the second control is selected; the second signal is generated by the second electronic device when the second physical button is pressed by a user.

[0032] In conjunction with the third aspect, in some embodiments, the second electronic device may be a game controller. A description of the game controller can be found in the description of the second electronic device provided in the first aspect above, and will not be repeated here.

[0033] In conjunction with the third aspect, in some embodiments, the processor can also be used to execute the function corresponding to the first control when a first signal is received through the first connection after the mapping relationship between the multiple controls and the multiple physical buttons of the second electronic device is established.

[0034] In conjunction with the third aspect, in some embodiments, the processor can also be specifically used to perform grayscale processing on the first user interface to obtain a first image, and then the processor can use an edge detection algorithm to identify the control boundaries contained in the first user interface from the first image.

[0035] In conjunction with the third aspect, in some embodiments, the multiple controls in the first user interface may include a first control, which may include multiple boundary pixels, and the multiple boundary pixels may include the first pixel. The higher the frequency with which the touch position corresponding to the first pixel in the first user interface is touched by the user, the higher the probability that the first pixel is identified as the boundary pixel of the first control.

[0036] In conjunction with the third aspect, in some embodiments, the processor can also be specifically used to, when the processor identifies the boundary of the first control in the first user interface from the first image using an edge detection method, employ an edge operator to calculate the gradient vector of each pixel in the first image. Then, the processor can use linear interpolation to compare the gradient value of the first gradient vector of the first pixel with the gradient values ​​of other pixels in the same direction as the first gradient vector of the first pixel. If the gradient value of the first pixel is the largest in the direction, then the processor can retain the gradient value of the first pixel and set the gradient values ​​of other pixels to zero. Next, the processor can set a first threshold; if the gradient value of the first pixel is greater than the first threshold, then the first pixel is retained, and all retained pixels constitute the boundary of the first control in the first user interface.

[0037] In conjunction with the third aspect, in some embodiments, when the processor executes the function corresponding to the first control, the processor displays a second user interface. This second user interface may be partially or entirely different from the aforementioned first user interface. Specifically, the second user interface may be a jump to a game application interface, the movement of a character within the game application interface, a change in the game scene, etc.

[0038] In conjunction with the third aspect, in some embodiments, when the first control in the first user interface is selected, the first control may be highlighted, or a cursor may be displayed in the area of ​​the first control, and / or the first control may be blinking. That is to say, this application does not limit how to prompt the user that the first control is selected.

[0039] In conjunction with the third aspect, in some embodiments, after the touchscreen displays a first user interface, a first floating control can be displayed on the first user interface. Specifically, the touchscreen can be used to detect a first user action (e.g., a click) performed on the first floating control.

[0040] Fourthly, embodiments of the present invention provide a computer storage medium storing a computer program, the computer program including executable instructions, which, when executed by a processor, cause the processor to perform operations corresponding to the methods provided in the first and second aspects. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the architecture of a communication system provided in an embodiment of this application;

[0042] Figure 2 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0043] Figure 3 This is a schematic diagram of the hardware structure of another electronic device provided in an embodiment of this application;

[0044] Figure 4 This is a software framework diagram of an electronic device provided in an embodiment of this application;

[0045] Figure 5 This is a software module architecture diagram of a communication system provided in an embodiment of this application;

[0046] Figure 6A This is a schematic diagram of a user interface provided in an embodiment of this application;

[0047] Figure 6B This is a schematic diagram of a user interface provided in an embodiment of this application;

[0048] Figure 6C This is a schematic diagram of a user interface provided in an embodiment of this application;

[0049] Figure 7 This is a schematic diagram of a user interface provided in an embodiment of this application;

[0050] Figure 8 This is a schematic diagram of a user interface provided in an embodiment of this application;

[0051] Figure 9 This is a schematic diagram of a user interface provided in an embodiment of this application;

[0052] Figure 10 This is a schematic diagram of a user interface provided in an embodiment of this application;

[0053] Figure 11 This is a schematic diagram of a user interface provided in an embodiment of this application;

[0054] Figure 12 This is a schematic diagram of a user interface provided in an embodiment of this application;

[0055] Figure 13 This is a schematic diagram of a user interface provided in an embodiment of this application;

[0056] Figure 14A This is a schematic diagram of a user interface provided in an embodiment of this application;

[0057] Figure 14BThis is a schematic diagram of a user interface provided in an embodiment of this application;

[0058] Figure 15 This is a flowchart of a key mapping method provided in an embodiment of this application;

[0059] Figure 16 This is a flowchart of a method for identifying a key area provided in an embodiment of this application;

[0060] Figure 17 This is a schematic diagram of identifying pixels in a nearby area provided in an embodiment of this application. Detailed Implementation

[0061] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to including one or more of the listed prominent features, any or all possible combinations. In the embodiments of this application, the terms “first” and “second” are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as “first” or “second” may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, “a plurality” means two or more.

[0062] This application provides a key mapping method that can be applied to the electronic device provided in this application. The electronic device can be a mobile phone, tablet computer, personal computer (PC), smart TV, or other electronic device; this application does not limit the specific type of electronic device. This method establishes a mapping relationship between virtual key elements in an application on the electronic device and physical keys (also called physical buttons) of a peripheral electronic device (e.g., a peripheral game controller). The user can then use the peripheral electronic device to control the application's keys to trigger the corresponding function. For example, taking a game application as an example, since the key area in a game application is separate from the game screen, meaning that the key area does not change with changes in the game screen, and there is a significant color contrast between the key area and the surrounding game screen, the electronic device can identify the key area in the game application using an image processing algorithm, according to the method in this application embodiment. Furthermore, to improve the accuracy of key area recognition, the electronic device can also combine the user's touch frequency on the game screen to correct the image processing algorithm. After identifying all key areas, each key area is numbered, and a cursor that the user can control to move appears on the electronic device interface. This cursor can move within the identified key area. Users can control the cursor to move across the identified button area, allowing them to select the virtual button icons that need to be matched with the game controller's physical buttons within the application interface. For example, when the cursor is on the first virtual game button, the user can press the first physical button on the game controller, thus establishing a mapping between the first physical button and the first virtual game button. Once all the virtual button icons in the game application have been matched with the physical buttons on the game controller, the mapping is complete. Users can then control the physical buttons on the game controller to trigger the virtual buttons in the game application, thereby activating the corresponding functions, such as character movement, skill casting, scene switching, and so on.

[0063] Compared with the prior art, the technical solution of this application can reduce the number of user operation steps in the process of matching game controller and game application buttons, making user operation simpler, matching more accurate, and greatly improving the user experience.

[0064] Below, we introduce some terms and concepts related to this application.

[0065] A user interface (UI) is the medium through which an application or operating system interacts and exchanges information with a user. It converts the internal form of information into a form that the user can understand. An application's user interface is written in source code using specific computer languages ​​such as Java or Extensible Markup Language (XML). This source code is parsed and rendered on the terminal device, ultimately presenting user-recognizable content such as images, text, buttons, and other controls.

[0066] In the binary counting of a computer, the leftmost bit of an unsigned integer is not used to represent positive or negative. Instead, it is combined with the following bits to represent an integer. Therefore, it is impossible to distinguish whether the number is positive or negative; it can only be a positive number. This is what an unsigned integer is.

[0067] Grayscale refers to representing objects using black tones; that is, using black as the base color and displaying images with different saturations of black. Each grayscale object has a brightness value ranging from 0% (white) to 100% (black) on the grayscale bar.

[0068] Noise refers to pixels or pixel blocks that appear extremely abruptly in an image, causing interference, making the image unclear, or affecting the observation of image details.

[0069] Gaussian noise refers to noise pixels whose probability density function follows a Gaussian distribution (also known as a normal distribution). In other words, if the amplitude of a noise follows a Gaussian distribution and its power spectral density is relatively uniformly distributed, it is called Gaussian noise.

[0070] Gaussian filtering is a linear smoothing filter suitable for eliminating Gaussian noise and widely used in image processing for noise reduction. Simply put, Gaussian filtering is a weighted average process applied to the entire image. The value of each pixel is obtained by weighted averaging of its own value and the values ​​of its neighboring pixels. Specifically, Gaussian filtering uses a template (or Gaussian kernel) to scan every pixel in the image, replacing the value at the center of the template with the weighted average gray value of the pixels in the neighborhood defined by the template. Gaussian filters are very effective for noise that consistently follows a normal distribution.

[0071] RGB refers to the color spectrum representing three channels: red (R), green (G), and blue (B). Electronic devices obtain various colors by varying these three color channels and superimposing them. This color representation standard covers almost all colors perceptible to human vision and is one of the most widely used color systems today.

[0072] The blur radius refers to the value by which a pixel expands outward during the Gaussian filtering process.

[0073] In vector calculus, the gradient at a point in a scalar field points in the direction of the fastest growth of the scalar field; the gradient strength is the maximum rate of change in that direction. For a single-variable real-valued function, the gradient is simply the derivative; or, for a linear function, the gradient is the slope of the line. Used to describe the inclination of a surface along a given direction, the numerical value of the gradient is sometimes also referred to as the gradient itself.

[0074] First, a communication system 10 provided in the embodiments of this application will be introduced.

[0075] like Figure 1 As exemplarily shown, the communication system 10 may include electronic device 100 and electronic device 200.

[0076] Electronic device 100 can be a mobile phone, tablet computer, PC, smart TV, or other electronic device. Specifically, electronic device 100 may have one or more of the following: a Bluetooth module and a WLAN module. Electronic device 100 can detect and scan nearby devices by transmitting signals through one or more of the Bluetooth module and WLAN module, enabling electronic device 100 to discover nearby devices (e.g., electronic device 200) using one or more wireless communication technologies such as Bluetooth or WLAN, and to establish wireless communication connections with nearby devices, transmitting data to nearby devices (e.g., electronic device 200) through one or more of the following wireless communication technologies: Bluetooth or WLAN. The Bluetooth module can provide one or more Bluetooth communication solutions, including classic Bluetooth (Bluetooth 2.1 standard) or Bluetooth Low Energy (BLE). The WLAN module can provide one or more WLAN communication solutions, including Wi-Fi Direct, Wi-Fi LAN, or Wi-Fi SoftAP.

[0077] Electronic device 200 may be a peripheral handle having a Bluetooth module, and / or a WLAN module, and / or a data cable interface. Electronic device 200 may receive or transmit wireless signals via one or more of the Bluetooth module and WLAN module. The Bluetooth module may provide one or more Bluetooth communication solutions, including classic Bluetooth or Bluetooth Low Energy. The WLAN module may provide one or more WLAN communication solutions, including Wi-Fi Direct, Wi-Fi LAN, or Wi-Fi SoftAP.

[0078] Electronic device 200 may also include a joystick button 201A, a start button 201B (marked with the letter S), function buttons A 201C, B 201D, C 201E, and D 201F. After the button mapping between electronic device 100 and electronic device 200 is completed, joystick button 201A can be used to control the directional buttons on the user interface of electronic device 100 for directional movement (e.g., up, down, etc.), and start button 201B can be used to start or stop electronic device 200. Function buttons A 201C, B 201D, C 201E, and D 201F can be mapped to various function buttons on the user interface of electronic device 100, and when the user presses these function buttons, the electronic device 100 can generate corresponding functional events.

[0079] like Figure 1 As shown, electronic device 200 can establish a first connection with electronic device 100. Specifically, the first connection can be one or more wireless communication connections such as Bluetooth, Wi-Fi Direct, or Wi-Fi SoftAP, or it can be a wired connection, such as a Universal Serial Bus (USB) connection. After electronic device 100 and electronic device 200 establish the first connection, electronic device 100 and the second electronic device can transmit data information to each other through the first connection.

[0080] It is understood that the structure of the electronic device 200 shown in the embodiments of this application does not constitute a specific limitation on the communication system 10. In other embodiments of this application, the electronic device 200 may have more or fewer buttons than shown; for example, the electronic device 200 may have multiple joystick buttons 201A. In other embodiments of this application, the buttons on the electronic device 200 may be located on the side, back, or other sides of the electronic device 200 that are not facing the user. This application does not impose any limitations on this.

[0081] It is understood that the illustrative structures shown in the embodiments of this application do not constitute a specific limitation on the communication system 10. In other embodiments of this application, the communication system 10 may include more or fewer devices than shown. For example, the communication system 10 may also include multiple mobile phones, or multiple different types of electronic devices, such as displays, tablets, PCs, etc., with communication functions. This application does not impose any limitations in this regard.

[0082] Next, we will introduce the exemplary electronic device 100 provided in the embodiments of this application.

[0083] Figure 2 A schematic diagram of the hardware structure of the electronic device 100 is shown.

[0084] Electronic device 100 may be a mobile phone, tablet computer, desktop computer, laptop computer, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, as well as cellular phone, personal digital assistant (PDA), augmented reality (AR) device, virtual reality (VR) device, artificial intelligence (AI) device, wearable device, in-vehicle device, smart home device and / or smart city device. The embodiments of this application do not impose any special restrictions on the specific type of electronic device.

[0085] Electronic device 100 may include processor 110, external memory interface 120, internal memory 121, universal serial bus (USB) interface 130, charging management module 140, power management module 141, battery 142, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0086] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0087] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0088] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0089] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0090] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0091] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C buses. The processor 110 can couple to the touch sensor 180K, charger, flash, camera 193, etc., through different I2C bus interfaces. For example, the processor 110 can couple to the touch sensor 180K through the I2C interface, enabling the processor 110 and the touch sensor 180K to communicate through the I2C bus interface, thereby realizing the touch function of the electronic device 100.

[0092] The I2S interface can be used for audio communication. In some embodiments, the processor 110 may include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to enable communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the I2S interface to enable the function of answering phone calls through a Bluetooth headset.

[0093] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via the PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface, enabling the function of answering phone calls through a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0094] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 via the UART interface to implement Bluetooth functionality. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the UART interface to enable music playback through Bluetooth headphones.

[0095] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI) and a display serial interface (DSI). In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to enable the electronic device 100 to capture images. The processor 110 and the display screen 194 communicate via the DSI interface to enable the electronic device 100 to display images.

[0096] The GPIO interface is configurable via software. It can be configured as a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 to a camera 193, a display screen 194, a wireless communication module 160, an audio module 170, a sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0097] USB port 130 is a USB standard compliant interface, specifically a Mini USB port, Micro USB port, USB Type-C port, etc. USB port 130 can be used to connect a charger to charge electronic device 100, and can also be used for data transfer between electronic device 100 and peripheral devices. It can also be used to connect headphones for audio playback. This interface can also be used to connect other electronic devices, such as AR devices.

[0098] It is understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0099] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device via the power management module 141.

[0100] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, display screen 194, camera 193, and wireless communication module 160, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be located in the same device.

[0101] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0102] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0103] The mobile communication module 150 can provide wireless communication solutions, including 2G / 3G / 4G / 5G, for use on the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device. In some embodiments, the wireless communication solution provided by the mobile communication module 150 enables the electronic device to communicate with devices (such as servers) in a network.

[0104] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio device (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 150 or other functional modules.

[0105] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2. In some embodiments, the electronic device 100 can use the Bluetooth module or WLAN module in the wireless communication module 160 to transmit signals to detect or scan devices near the electronic device 100, establish wireless communication connections with nearby devices, and transmit data. The Bluetooth module can provide solutions for one or more Bluetooth communication methods, including classic Bluetooth (Bluetooth 2.1 standard) or Bluetooth Low Energy. The WLAN module can provide solutions for one or more WLAN communication methods, including Wi-Fi Direct, Wi-Fi LAN, or Wi-Fi SoftAP.

[0106] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0107] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0108] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0109] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0110] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0111] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0112] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 100 selects a frequency, the DSP performs Fourier transforms on the frequency energy.

[0113] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0114] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0115] Internal memory 121 may include one or more random access memory (RAM) and one or more non-volatile memory (NVM).

[0116] Random access memory can include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM, for example, fifth generation DDR SDRAM is generally called DDR5 SDRAM), etc.

[0117] Non-volatile memory can include disk storage devices and flash memory.

[0118] Flash memory can be classified according to its operating principle, including NOR FLASH, NAND FLASH, 3D NAND FLASH, etc.; according to the level of the storage cell, including single-level cell (SLC), multi-level cell (MLC), triple-level cell (TLC), quad-level cell (QLC), etc.; and according to the storage specification, including universal flash storage (UFS) and embedded multimedia card (eMMC), etc.

[0119] The random access memory can be directly read and written by the processor 110. It can be used to store executable programs (such as machine instructions) of the operating system or other running programs, as well as user and application data.

[0120] Non-volatile memory can also store executable programs and user and application data, and can be pre-loaded into random access memory for direct reading and writing by the processor 110.

[0121] The external memory interface 120 can be used to connect to external non-volatile memory, thereby expanding the storage capacity of the electronic device 100. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to perform data storage functions. For example, music, video, and other files can be stored in the external non-volatile memory.

[0122] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0123] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0124] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or make hands-free calls through the speaker 170A.

[0125] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the electronic device 100 answers a telephone call or voice message, the receiver 170B can be brought close to the ear to listen to the voice.

[0126] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C. Electronic device 100 may have at least one microphone 170C. In some embodiments, electronic device 100 may have two microphones 170C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, electronic device 100 may also have three, four, or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0127] The 170D headphone jack is used to connect wired headphones. The 170D headphone jack can be a USB 130 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0128] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A can be disposed on display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may include at least two parallel plates with conductive material. When force is applied to pressure sensor 180A, the capacitance between the electrodes changes. Electronic device 100 determines the pressure intensity based on the change in capacitance. When a touch operation is applied to display screen 194, electronic device 100 detects the intensity of the touch operation based on pressure sensor 180A. Electronic device 100 can also calculate the touch position based on the detection signal from pressure sensor 180A. In some embodiments, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation commands. For example, when a touch operation with an intensity less than a first pressure threshold is applied to the SMS application icon, a command to view an SMS is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to the SMS application icon, a command to create a new SMS is executed.

[0129] The gyroscope sensor 180B can be used to determine the motion attitude of the electronic device 100. In some embodiments, the gyroscope sensor 180B can determine the angular velocity of the electronic device 100 about three axes (i.e., the x, y, and z axes). The gyroscope sensor 180B can be used for image stabilization. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the shake of the electronic device 100, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to counteract the shake of the electronic device 100 by moving in the opposite direction, thus achieving image stabilization. The gyroscope sensor 180B can also be used in navigation and motion-sensing game scenarios.

[0130] The barometric pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates altitude using the air pressure value measured by the barometric pressure sensor 180C to assist in positioning and navigation.

[0131] The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip cover. In some embodiments, when the electronic device 100 is a flip phone, the electronic device 100 can detect the opening and closing of the flip cover using the magnetic sensor 180D. Then, based on the detected opening and closing state of the cover or the flip cover, features such as automatic flip unlocking can be set.

[0132] The 180E accelerometer can detect the magnitude of acceleration of electronic device 100 in various directions (typically three axes). When electronic device 100 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the posture of electronic devices and applied to applications such as screen orientation switching and pedometers.

[0133] A distance sensor 180F is used to measure distance. Electronic device 100 can measure distance via infrared or laser. In some embodiments, during a shooting scene, electronic device 100 can utilize the distance sensor 180F to measure distance for rapid focusing.

[0134] The proximity sensor 180G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The LED may be an infrared LED. The electronic device 100 emits infrared light outward through the LED. The electronic device 100 uses the photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100. The electronic device 100 may use the proximity sensor 180G to detect when a user holds the electronic device 100 close to their ear for a call, so as to automatically turn off the screen to save power. The proximity sensor 180G can also be used in holster mode and pocket mode for automatic unlocking and locking of the screen.

[0135] The ambient light sensor 180L is used to sense the brightness of ambient light. The electronic device 100 can adaptively adjust the brightness of the display screen 194 based on the sensed ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also work with the proximity sensor 180G to detect whether the electronic device 100 is in a pocket to prevent accidental touches.

[0136] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can utilize the characteristics of the collected fingerprints to achieve fingerprint unlocking, accessing application locks, taking photos with fingerprints, answering calls with fingerprints, etc.

[0137] Temperature sensor 180J is used to detect temperature. In some embodiments, electronic device 100 uses the temperature detected by temperature sensor 180J to execute a temperature handling strategy. For example, when the temperature reported by temperature sensor 180J exceeds a threshold, electronic device 100 performs thermal protection by reducing the performance of a processor located near temperature sensor 180J to reduce power consumption. In other embodiments, when the temperature is below another threshold, electronic device 100 heats battery 142 to prevent abnormal shutdown of electronic device 100 due to low temperature. In still other embodiments, when the temperature is below yet another threshold, electronic device 100 boosts the output voltage of battery 142 to prevent abnormal shutdown due to low temperature.

[0138] Touch sensor 180K, also known as a "touch device," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touchscreen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of electronic device 100, in a different position than display screen 194.

[0139] The bone conduction sensor 180M can acquire vibration signals. In some embodiments, the bone conduction sensor 180M can acquire vibration signals from the vibrating bone segments of the human vocal cords. The bone conduction sensor 180M can also contact the human pulse to receive blood pressure signals. In some embodiments, the bone conduction sensor 180M can also be incorporated into headphones to form bone conduction headphones. The audio module 170 can parse the voice signals from the vibrating bone segments of the vocal cords acquired by the bone conduction sensor 180M to realize voice functionality. The application processor can parse heart rate information from the blood pressure signals acquired by the bone conduction sensor 180M to realize heart rate detection functionality.

[0140] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. Electronic device 100 can receive button input and generate key signal inputs related to user settings and function control of electronic device 100.

[0141] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can correspond to touch operations performed on different applications (such as taking photos, playing audio, etc.). Motor 191 can also correspond to different vibration feedback effects for touch operations performed on different areas of the display screen 194. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0142] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0143] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to make contact with and separate from the electronic device 100. The electronic device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 simultaneously. The multiple cards can be of the same or different types. The SIM card interface 195 is also compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to realize functions such as calls and data communication. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0144] Figure 3 The hardware structure of the electronic device 200 provided in this application is illustrated by way of example.

[0145] like Figure 3As shown, the electronic device 200 may include a button 201, a processor (central processing unit, CPU) 202, a memory 203, a bus 204, an input / output interface 205, a motor 206, an indicator light 207, an audio module 208, a sensor module 209, a communication interface 210, a wireless communication module 211, a power management module 212, and an antenna 3. The sensor module 209 may include a pressure sensor 209A, an angle sensor 209B, a gravity sensor 209C, a gyroscope sensor 209D, an accelerometer 209E, etc. The communication interface 210 may include a USB interface 210A and a wireless communication interface 210B, etc. The wireless communication module 211 may include a Bluetooth communication module 211A and a Wi-Fi communication module 211B, etc. The processor 202, the communication interface 210, the wireless communication module 211, and the power management module 212 can be connected via the bus 204 or other means. Figure 3 Take the connection via bus 204 as an example.

[0146] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 200. In other embodiments of this application, the electronic device 200 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0147] Button 201 may include, for example Figure 1 The diagram shows the start button 201B, the joystick button 201A, and various function buttons. Button 201 can be a mechanical button. Electronic device 100 can receive button signals from electronic device 200 and generate touch events for the corresponding application buttons on the display screen of electronic device 100.

[0148] Processor 202 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0149] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0150] The processor 202 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 202 is a cache memory. This memory can store instructions or data that the processor 202 has just used or that are used repeatedly. If the processor 202 needs to use the instruction or data again, it can directly retrieve it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 202, and thus improves the efficiency of the system.

[0151] In some embodiments, the processor 202 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0152] The memory 203 may include one or more random access memory (RAM) and one or more non-volatile memory (NVM).

[0153] Random access memory can include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM, for example, fifth generation DDR SDRAM is generally called DDR5 SDRAM), etc.

[0154] Non-volatile memory can include disk storage devices and flash memory.

[0155] Flash memory can be classified according to its operating principle, including NOR FLASH, NAND FLASH, 3D NAND FLASH, etc.; according to the level of the storage cell, including single-level cell (SLC), multi-level cell (MLC), triple-level cell (TLC), quad-level cell (QLC), etc.; and according to the storage specification, including universal flash storage (UFS) and embedded multimedia card (eMMC), etc.

[0156] Random access memory can be directly read and written by the processor 202. It can be used to store executable programs (such as machine instructions) of the operating system or other running programs, as well as user and application data.

[0157] Non-volatile memory can also store executable programs and user and application data, and can be pre-loaded into random access memory for direct reading and writing by the processor 202.

[0158] Motor 206 can generate vibration feedback. For example, motor 206 can be used for touch vibration feedback, and different vibration feedback effects can be corresponding to different touch operations (e.g., pressing) of different buttons 201 (e.g., start button, function button, etc.). Motor 206 can also correspond to different vibration feedback effects for touch operations of different buttons 201. In some embodiments, the touch vibration feedback effect can also be customized.

[0159] The indicator 207 can be an indicator light, used to indicate charging status, power changes, or touch operation of button 201, etc.

[0160] The audio module 208 is used to convert digital audio information into analog audio signal output, and also to convert analog audio input into digital audio signal. The audio module 208 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 208 may be located in the processor 202, or some functional modules of the audio module 208 may be located in the processor 202.

[0161] Pressure sensor 209A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 209A can be disposed at the bottom of button 201. There are many types of pressure sensors 209A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may include at least two parallel plates with conductive material. When force is applied to pressure sensor 209A, the capacitance between the electrodes changes. Electronic device 200 determines the pressure intensity based on the change in capacitance. When a touch operation (e.g., pressing) is applied to button 201, electronic device 200 detects the intensity of the touch operation based on pressure sensor 209A. Electronic device 200 can also calculate the touch position based on the detection signal from pressure sensor 209A. In some embodiments, touch operations applied to the same touch position but with different durations can correspond to different operation commands. For example, when electronic device 200 is in operation, if a touch operation with a duration less than a first duration threshold is applied to the start button, a command to put electronic device 200 into sleep mode is executed. When a touch operation with a duration greater than or equal to a first duration threshold is applied to the start button, an instruction to shut down the electronic device 200 is executed.

[0162] Angle sensor 209B can be used to detect angles. In its implementation, angle sensor 209B may have a hole in the center to engage with a corresponding mechanical shaft. Angle sensor 102 counts once every 1 / 16th of a revolution of the mechanical shaft. The count increases when rotating in one direction and decreases when the direction of rotation changes. The count is related to the initial position of angle sensor 102; when the angle sensor is initialized, its count value is set to 0. If necessary, the angle sensor can be reset using programming.

[0163] The gravity sensor 209C can be used to collect gravitational acceleration data of the electronic device 200 to determine the motion state of the electronic device 200. The gravity sensor 209C can be used in scenarios such as motion-sensing games.

[0164] The gyroscope sensor 209D can be used to determine the motion posture of the electronic device 200 and send the motion posture-related signals to the electronic device 100, thereby enabling the corresponding control element icons on the display screen of the electronic device 100 to display the same motion posture as the electronic device 200. In some embodiments, the gyroscope sensor 209D can determine the angular velocity of the electronic device 200 around three axes (i.e., the x, y, and z axes). The gyroscope sensor 180B can be used in scenarios such as motion-sensing games and racing games.

[0165] Accelerometer 209E can detect the magnitude of acceleration of electronic device 200 in various directions (generally three axes). Therefore, accelerometer 209E can be used to detect the motion information of electronic device 200. When electronic device 200 is stationary, accelerometer 209E can also detect the magnitude and direction of gravity.

[0166] USB interface 210A is an interface compliant with USB standards, specifically a Mini USB interface, Micro USB interface, USB Type-C interface, etc. USB interface 210A can be used to connect a charger to charge electronic device 200, and can also be used for data transfer between electronic device 200 and peripheral devices. This interface can also be used to connect other electronic devices, such as AR devices. In some embodiments, USB interface 210A can also be a plug-and-play (OTG) interface, mainly used for connecting various different devices and exchanging data. It can provide USB interface accessories for smart terminals to enrich their functionality. The OTG interface can connect electronic device 200 to peripheral devices via an OTG data cable with a USB interface on one end and a Type-C interface on the other, and can transfer data through this connection method.

[0167] The wireless communication interface 210B is an interface that conforms to a wireless communication protocol, specifically an 802.11 wireless interface, etc. The wireless communication interface 210B can be used to establish a wireless connection between the electronic device 200 and peripheral devices, and to transmit data between the devices through this wireless connection.

[0168] The wireless communication processing module 211 may include one or more of the Bluetooth communication processing module 211A and the WLAN communication processing module 211B. It can be used to listen to signals emitted by other devices (such as electronic devices 100), such as probe requests, scan signals, etc., and can send response signals, such as probe responses, scan responses, etc., so that other devices (such as electronic devices 100) can discover electronic devices 200 and establish wireless communication connections with other devices (such as electronic devices 100), and communicate with other devices (such as electronic devices 100) through one or more wireless communication technologies such as Bluetooth or WLAN.

[0169] In other embodiments, one or more of the Bluetooth communication processing module and the WLAN communication processing module may also transmit signals, such as broadcasting Bluetooth signals or beacon signals, so that other devices (e.g., electronic device 100) can discover electronic device 200 and establish wireless communication connections with other devices (e.g., electronic device 100) and communicate with other devices (e.g., electronic device 100) through one or more wireless communication technologies such as Bluetooth or WLAN.

[0170] Antenna 3 is used to transmit and receive electromagnetic wave signals, and can cover one or more communication frequency bands. In some other embodiments, the antenna can be used in conjunction with a tuning switch.

[0171] In other embodiments, the electronic device 200 may also include multiple antennas, which is not a limitation of this application.

[0172] The power management module 212 may include a battery and charging management module. The power management module 212 can supply power to the processor 202, memory 203, and wireless communication module 211, etc. The power management module 212 can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 212 may also be located within the processor 202. In other embodiments, the charging management module within the power management module 212 may be located in a different device.

[0173] The software system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This embodiment of the invention uses a layered architecture. Taking the system as an example, the software architecture of electronic device 100 is illustrated.

[0174] Figure 4 This is a software structure block diagram of the electronic device 100 according to an embodiment of this application. For example... Figure 4 As shown, the software structure of the electronic device 100 may include: an application layer (APP), an application framework layer (FWK), and an Android runtime layer (…). runtime and system libraries, kernel layer and hardware layer.

[0175] The application layer can include a series of application packages.

[0176] like Figure 4 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and SMS.

[0177] In this embodiment, the application layer may further include game applications and game controller applications.

[0178] A game application is an application on electronic devices such as smartphones and tablets that provides entertainment for users. This application does not limit the name of the application. That is to say, the game application can be any game application available to users on the market that they can obtain and control, for example... However, this application does not impose any restrictions on this.

[0179] The gamepad application can be used to manage and configure applications on the electronic device 200 (e.g., a peripheral gamepad). For example, the gamepad application can be used to set and / or adjust parameters such as button sensitivity, button icon transparency, and click rate. It can also be used to identify button areas on the application user interface of the electronic device 100 that require button mapping. After the button area identification step is completed, the gamepad application can respond to the user's operation, establish mapping information between the virtual button icons that need to be mapped and the physical buttons on the electronic device 200, and save the mapping information in the internal storage of the electronic device 100 and / or a cloud server.

[0180] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0181] like Figure 4As shown, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, patch package, etc.

[0182] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0183] Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0184] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0185] The phone manager is used to provide communication functions for electronic device 100. For example, it manages call status (including connection and disconnection).

[0186] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0187] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of download completion or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting alert sounds, vibrating head-mounted displays, and flashing indicator lights.

[0188] Runtime includes core libraries and a virtual machine. The runtime is responsible for scheduling and management of the Android system.

[0189] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0190] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0191] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), patching engines, etc.

[0192] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0193] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0194] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0195] A 2D graphics engine is a graphics engine for 2D drawing.

[0196] The kernel layer is the layer between hardware and software. At a minimum, the kernel layer includes display drivers, camera drivers, audio drivers, sensor drivers, as well as WLAN and Bluetooth capabilities and basic communication protocols.

[0197] The following example, using a scene of capturing a photograph, illustrates the workflow of the software and hardware of the electronic device 100.

[0198] When touch sensor 180K receives a touch operation, a corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, timestamp of the touch operation, etc.). The raw input event is stored in the kernel layer. The application framework layer retrieves the raw input event from the kernel layer and identifies the control corresponding to the input event. Taking the touch operation as an example, where the corresponding control is the camera application icon, the camera application calls the interface of the application framework layer to launch the camera application, and then calls the kernel layer to launch the camera driver, capturing still images or videos through camera 193.

[0199] The software modules of the communication system 10 are described below.

[0200] Figure 5 This is a software architecture block diagram of communication system 10. (Example) Figure 5 As shown, the software architecture of the communication system 10 may include software modules of electronic device 100 and electronic device 200. The software modules of electronic device 100 may include a processing module 510, a communication module 520, and a display module 530. Electronic device 200 may include a communication module 540 and a processing module 550.

[0201] The processing module 510 includes a button recognition module 511, a button mapping module 512, and a button management module 513. The button recognition module 511 identifies the button icon area on the current user interface of the electronic device 100 after establishing a communication connection with the electronic device 200 and launching an application requiring button mapping. The button management module 513 manages and adjusts the game controller's parameter configurations, such as button sensitivity, button icon transparency, and click rate. The button management module 513 also obtains the coordinate positions of the button icons on the user interface of the electronic device 100 and the key values ​​corresponding to the physical buttons touched by the user on the electronic device 200, generating and saving the mapping information between the aforementioned button icon coordinate positions and the corresponding key values ​​of the physical buttons.

[0202] In other embodiments, after establishing a communication connection with electronic device 200 and launching an application that requires key mapping, electronic device 100 detects a user's touch operation on an icon used to activate the key area recognition module (e.g., clicking the "Start Matching" icon). In response to this operation, key recognition module 511 recognizes the key icon area on the current user interface of electronic device 100.

[0203] The communication module 520 may include managing USB-based wired connections and / or managing wireless connections based on one or more wireless communication technologies, including Bluetooth and / or WLAN. The Bluetooth (BT) module may provide solutions for one or more Bluetooth communication technologies, including Bluetooth Classic (Bluetooth 2.1) or Bluetooth Low Energy (BLE). The WLAN module may provide solutions for one or more WLAN communication technologies, including Wi-Fi Direct, Wi-Fi LAN, or Wi-Fi SoftAP.

[0204] The display module 530 can be used to display a user interface, such as images or videos, on the electronic device 100 so that users can interact with the electronic device 100.

[0205] The communication module 540 is a module used by the electronic device 200 to communicate with other devices. For details, please refer to the description of the communication module 520 above, which will not be repeated here.

[0206] As shown in the figure, electronic devices 200 and 100 can exchange data through a first connection established between communication module 520 and communication module 540. This first connection can be referred to in the preceding description. Figure 1 The description in the text will not be repeated here.

[0207] The processing module 550 can be used for signal processing of the game controller. Specifically, for example, in response to a user's touch operation (e.g., pressing) on ​​the electronic device 200, the electronic device 200 can receive a physical signal (e.g., a pressure signal). Then, the processing module 550 can convert the physical signal into an electrical signal through the sensors (e.g., pressure sensor 209A) in the aforementioned sensor module 209. The processing module 550 can also collect and process the data in the aforementioned sensor module 209. For example, when the electronic device 200 shifts to the left, the gravity sensor 209C in the sensor module 209 detects the gravitational acceleration moving downwards from the left. The processing module 550 converts this physical signal into an electrical signal, processes it accordingly (e.g., processes it to a suitable frequency), and sends it to the electronic device 100 via the communication module 540. After receiving and processing the data via the communication module 520, the electronic device 100 displays the corresponding touch event on the display module 530 (e.g., controllable icon elements on the electronic device 100 also shift to the left accordingly).

[0208] The following describes an exemplary user interface for application menus on electronic device 100.

[0209] Figure 6A An exemplary user interface 30 is shown.

[0210] The user interface 30 may include: a status bar 301, a tray 302 with icons of commonly used applications, a calendar indicator 303, a page indicator 304, and other application icons, etc.

[0211] The status bar 301 may include: one or more signal strength indicators 301A for mobile communication signals (also known as cellular signals), one or more signal strength indicators 301B for wireless fidelity (Wi-Fi) signals, and a battery status indicator 301C.

[0212] The tray 302, which contains icons for commonly used applications, can display: camera icon 302A, contacts icon 302B, phone icon 302C, and message icon 302D.

[0213] The calendar indicator 303 can be used to indicate the current time, such as the date, day of the week, hour and minute information, etc.

[0214] Page indicator 304 can be used to indicate which application on which page the user is currently viewing. Users can swipe left or right on the area containing other application icons to browse application icons on other pages.

[0215] Other application icons can be, for example: music icon 305, calculator icon 306, game icon 307, and settings icon 308.

[0216] In some embodiments, Figure 6A The user interface 30 shown in the example can be the main screen.

[0217] In some other embodiments, the electronic device may also include a home screen button. This home screen button can be a physical button or a virtual button. The home screen button can receive user commands to return the currently displayed UI to the home screen, allowing the user to easily access the home screen at any time. Specifically, the command may be a single press of the home screen button, a double press within a short period, or a long press for a predetermined time. In other embodiments of this application, the home screen button may also integrate a fingerprint reader, enabling fingerprint collection and recognition when the user presses the home screen button.

[0218] Understandable, Figure 6A The user interface on the electronic device 100 is merely shown as an example and should not be construed as limiting the embodiments of this application.

[0219] The following describes the application scenarios involved in this application and some embodiments of the user interface implemented on the electronic device 100.

[0220] Figure 6A , Figure 6B , Figure 6C and Figure 7 The user interface for establishing a communication connection between electronic device 100 and electronic device 200 is shown.

[0221] This application embodiment takes the communication between electronic device 100 and electronic device 200 via Bluetooth as an example. Figure 6A An example is shown of an operation to enable Bluetooth on an electronic device 100.

[0222] like Figure 6AAs shown, when a downward swipe gesture is detected on the status bar 301, the electronic device 100 can display the user interface 31 in response to the gesture. The user interface 31 may include a window 311 and some or all of the interface elements (such as controls, icons, text, etc.) that are the same as those in the aforementioned user interface 30. The window 311 may display a "Bluetooth" switch control 312, and may also display switch controls with other functions (such as Wi-Fi controls, flashlight controls, location information controls, game controller controls 313, etc.). When an operation (such as a touch operation) is detected on the Bluetooth switch control 312 in the window 311, the electronic device 100 can turn on the Bluetooth function in response to the operation. That is, the user can make a downward swipe gesture on the status bar 301 to open the window 311, and can click the "Bluetooth" switch control 312 in the window 311 to easily turn on Bluetooth. After Bluetooth is turned on, the electronic device 100 can discover nearby devices through Bluetooth communication technology.

[0223] Figure 6B Another method for enabling Bluetooth is illustrated. When a user touches (e.g., clicks) on the settings icon 308 is detected, the electronic device 100 can display the user interface 32 in response to the action.

[0224] The user interface 32 may include one or more settings, which may include: airplane mode settings, Wi-Fi settings, Bluetooth settings 321, mobile network settings, game controller settings 322, do not disturb mode settings, display and brightness settings, Huawei account settings, etc.

[0225] Each setting item on the user interface 32 has a corresponding title. For example, the title of the flight mode setting item is "Flight Mode", the title of the Wi-Fi setting item is "Wi-Fi", the title of the Bluetooth setting item 321 is "Bluetooth", the title of the mobile network setting item is "Mobile Network", the title of the game controller setting item 322 is "Game Controller", the title of the Do Not Disturb mode setting item is "Do Not Disturb", the title of the display and brightness setting item is "Display and Brightness", and the title of the Huawei account setting item is "Huawei Account". Each setting item can be used to listen for operations that trigger the display of the corresponding setting item's settings (such as touch operations). In response to this operation, the electronic device 100 can open the user interface used to display the settings of the corresponding setting item.

[0226] In other embodiments, the user interface 32 may include setting entries, such as "Assistant," "Biometrics and Password," etc. In other embodiments, the setting entries in the user interface 32 may also have corresponding text descriptions. The user interface 32 may also have fewer entries, and the titles corresponding to the setting entries may be different. The presentation of each setting item may include icons and / or text. This application does not impose any limitations on this.

[0227] In the context of user interface 32, a touch operation (e.g., a click) by the user on Bluetooth setting item 321 is detected. In response to this operation, electronic device 100 can enable Bluetooth. Once Bluetooth is enabled, electronic device 100 can discover nearby devices through Bluetooth communication technology.

[0228] In other embodiments, the electronic device 100 may also discover nearby devices through communication technologies such as Wi-Fi Direct (e.g., Wi-Fi p2p), Wi-FisoftAP, and Wi-Fi LAN, and this application does not limit this.

[0229] Figure 6C An exemplary user interface 33 for Bluetooth settings is shown. When a touch operation (e.g., a click) by the user on the Bluetooth switch control 312 or the Bluetooth settings entry 312 is detected, the electronic device 100 displays, in response to the operation, a user interface 33 for Bluetooth settings. Figure 6C The user interface shown is 33.

[0230] User interface 33 may include the status bar 301 shown in the aforementioned user interface 30, which may refer to the aforementioned... Figure 6A The description will not be repeated here. The user interface 33 may also include a current page indicator 331, a Bluetooth status control 332, a game controller 200 device option entry 323, and other interface elements (such as icons, controls, text, etc.).

[0231] The current page indicator 331 can be used to indicate the current page. For example, the text message "Bluetooth" can be used to indicate that the current page is used to display the Bluetooth settings main interface. It is not limited to text; the current page indicator 311 can also be an icon.

[0232] The Bluetooth status control control 332 can be used to listen for touch operations (such as clicks) applied to the control. In response to this operation, the electronic device 100 can turn the Bluetooth function on or off.

[0233] The device option entry 323 of the game controller 200 can be used to listen for touch operations (such as clicks) performed on that entry. In response to this operation, the electronic device 100 can establish a Bluetooth wireless communication connection with the game controller 200.

[0234] In other embodiments, the user interface 33 may display more device option entries, such as mobile phone device option entries, tablet computer device option entries, and so on. This application does not limit this.

[0235] Figure 7 An example is shown of the user interface 40 displayed by electronic device 100 when electronic device 100 and electronic device 200 successfully establish a Bluetooth communication connection.

[0236] User interface 40 may include a Bluetooth icon 401, a gamepad icon 402, a gamepad hover control 403, and other features as described above. Figure 6A The user interface 30 shown contains identical or all of the same interface elements (such as controls, icons, text content, etc.). Specifically, the Bluetooth icon 401 indicates to the user that electronic device 100 and electronic device 200 have established a wireless communication connection via Bluetooth, and the gamepad icon 402 indicates that electronic device 100 and electronic device 200 have successfully established a communication connection. The gamepad hover control 403 can be used to listen for touch operations applied to the control; in response to such operations, electronic device 100 can display a user interface for configuring the gamepad.

[0237] In some other embodiments, the handle hover control 402 may also display text information, such as "game controller", and this application does not limit this.

[0238] After electronic device 100 and electronic device 200 successfully establish a communication connection, electronic device 100 can display a function debugging interface for electronic device 200.

[0239] In some embodiments, the function debugging interface can be used to display one or more function debugging options for electronic device 200. These function debugging options can be used to set and modify parameters related to electronic device 200 or electronic device 100. Electronic device 100 can store the settings and modifications of these parameters so that when electronic device 100 and electronic device 200 re-establish a communication connection, the parameter settings can be directly used without requiring the user to manually debug again. The parameters that can be used for user debugging include, but are not limited to, the following options: click mode, associated mouse, wheel mode, gesture mode, etc., and this application does not limit this.

[0240] Figure 8 , Figure 9 , Figure 10 , Figure 11 An example is shown of an implementation method for functional debugging of electronic device 200 by electronic device 100.

[0241] like Figure 8As exemplarily shown, the electronic device 100 can display a touch operation (e.g., a click) performed by a user on the game controller control 313, the game controller settings entry 322, or the game controller hover control 403. Figure 8 The exemplary function debugging user interface 41 is shown.

[0242] The user interface 41 may include a current page indicator 411, a button transparency setting entry 412, a button sensitivity setting entry 413, a combo mode setting entry 414, a save control 415, a cancel control 416, and so on.

[0243] The current page indicator 411 can be used to indicate the current page. For example, the text "Gamepad" can be used to indicate that the current page displays the main interface for gamepad settings. It is not limited to text; the current page indicator 411 can also be an icon.

[0244] The button transparency setting entry 412 may include a corresponding title "Button Transparency," text information "25," and a button transparency adjustment control. This text information can change based on an operation performed on the button transparency adjustment control (e.g., dragging). For example, if the operation is dragging to the right, the number in the text information can increase; if the operation is dragging to the left, the number in the text information can decrease. The button transparency adjustment control in the button transparency setting entry 412 can be used to listen for operations performed on the control (e.g., dragging), and in response to this operation, the electronic device 100 can display the corresponding transparency level of the virtual button icon on the user interface.

[0245] The button sensitivity setting entry 413 may include a corresponding title "Button Sensitivity," text information "46," and a button sensitivity adjustment control. This text information can change based on the operation performed on the button sensitivity adjustment control (e.g., dragging). For example, if the operation is dragging to the right, the number in the text information can increase; if the operation is dragging to the left, the number in the text information can decrease. The button sensitivity adjustment control in the button sensitivity setting entry 413 can be used to monitor the operation performed on the control (e.g., dragging), and in response to this operation, can be used to adjust the actuation force of the physical buttons on the electronic device 200.

[0246] The combo rate setting entry 414 may include a corresponding title "Combo Rate," text information "46," and a combo rate switch control 414A and a combo rate adjustment control 414B. The text information can change based on the operation performed on the button sensitivity adjustment control (e.g., dragging). For example, if the operation performed on the combo rate adjustment control 414B is dragging to the right, the number in the text information can increase. If the operation performed on the combo rate adjustment control 414B is dragging to the left, the number in the text information can decrease. The combo rate switch control 414A in the combo rate setting entry 414 can be used to listen for touch operations (e.g., clicks) performed on the control; in response to this operation, the electronic device 100 activates the combo mode. The combo rate adjustment control 414B in the button sensitivity setting entry 413 can be used to listen for operations performed on the control (e.g., dragging); in response to this operation, the electronic device 100 can set the time interval for each combo mode trigger.

[0247] In other embodiments, the user interface 41 may add or remove settings items, and the titles corresponding to the settings items may also be different. The presentation of each setting item may include icons and / or text. This application does not impose any limitations on this.

[0248] In other embodiments, the setting operation can be performed in other scenarios. For example, in a game application scenario, Figure 9 and Figure 10 An example of setting up operations in a game application scenario is shown.

[0249] like Figure 9 As shown, in response to a touch operation (e.g., a click) on the game icon 307, the electronic device 100 displays a user interface 60 (also referred to as a first user interface). This user interface 60 is an exemplary user interface of a game application shown in an embodiment of this application.

[0250] like Figure 9 As shown, the user interface 60 may include: text information and graphic elements, a game character 601, a directional button area 602, a function button area 603, and a gamepad floating control 403. Among them:

[0251] Text information and graphic elements may include text information such as "Adventure Game" to indicate the current game name to the user, text information such as "Stamina: 48" to indicate the user's current data in the game, and other suggestive text information such as "Level 1-9 Dungeon", "Perform Mission", "Mission", "Daoxiang Village: Recommended Combat Power 1440", etc.

[0252] The game character 601 is the main entity controlled by the user in the game, and can respond to touch operations (such as clicking) on ​​the directional button controls and / or skill button controls in the user interface 60. The game character 601 can respond to touch operations on the directional buttons to perform movement actions, and can also respond to touch operations on the skill buttons to release corresponding skills, etc.

[0253] The directional button area 602 may include an up button icon 602A, a right button icon 602B, a down button icon 602C, and a left button icon 602D. The directional button area 602 can be used to receive touch operations (e.g., clicks) performed by the user on the buttons in this area. In response to this operation, the electronic device 100 can display the movement of the game character icon 601 in the corresponding direction (e.g., up, down, left, right, etc.). For example, if the electronic device 100 detects a touch operation (e.g., a long press) performed by the user on the up button icon 602A in the directional button area 602, then the electronic device 100 displays a scene where the game character icon 601 moves upwards in the game scene.

[0254] In some embodiments, the directional button area 602 may include more or fewer directional buttons than illustrated. For example, the directional button area 602 may also include a diagonal upper-right directional button, a diagonal lower-right directional button, and so on. In other embodiments, the button area 602 may be a circular icon used to listen for user touch operations on the circular icon (e.g., long-pressing the circular icon and dragging it in any direction). In response to this operation, the electronic device 100 displays a user interface showing the game character icon 601 moving in any direction within the game scene. This application does not impose any limitations on this.

[0255] For example, the skill button area 603 may include an equipment button icon 603A, a map button icon 603B, a basic attack button icon 603C, and a special move button icon 603D. The function area buttons 603 can be used to receive touch operations (e.g., clicks) performed by the user on the buttons in that area. In response to this operation, the electronic device 100 can display the user interface for the corresponding skill.

[0256] In other embodiments, the skill button area may include more or fewer skill buttons than shown in the figure. For example, the skill button area 603 may also include a "Set" skill button, a "Collect Equipment" skill button, etc., and this application does not limit this.

[0257] The controller hover control 403 can be referenced from the above. Figure 7 The description of the floating control 403 on the middle handle will not be repeated here.

[0258] It is understandable that the user interface 60 of this game application can be a scenario of other game applications; for example, the user interface 60 can be the aforementioned Figure 4 The user interface of the game application described in the application layer.

[0259] Not limited to this, in another possible embodiment, the user can also obtain the user interface 60 of the game application through other means, for example, the user can... The system retrieves the user interface of a game application by searching for a specific game mini-program. It's a social networking application that also provides users with various types of mini-programs (such as shopping, games, news, etc.). The mini-programs mentioned above are... This application provides an application that can be used without downloading, allowing users to experience mini-programs developed by developers through QR codes, searches, and other methods. This application does not impose any restrictions on this.

[0260] Figure 10 The diagram illustrates a settings main interface displayed by an electronic device 100 in an exemplary gaming application scenario.

[0261] like Figure 10 As shown, in response to a touch operation (e.g., a click) on the handle hover control 403 (also referred to as the first hover control), the electronic device 100 can display a user interface 61. The user interface 61 may include a function debugging bar 611, as well as the aforementioned... Figure 9 The user interface 60 shown displays interface elements (such as controls, icons, text content, etc.). The function debugging bar 611 may include a game identification icon 611A, a show / hide icon 611B, a settings icon 611C, and a problem feedback icon 661D.

[0262] The game recognition icon 611A can be used to listen for user touch operations via the icon. In response to this operation, the electronic device 100 can recognize the directional button area and skill button area in the user interface 60. Subsequent embodiments will describe in detail the button recognition steps provided by the electronic device 100, and will not be repeated here.

[0263] The show / hide icon 611B can be used to listen to the user's touch operation through the icon. In response to the operation, the electronic device 100 can show or hide the virtual button icon in the user interface.

[0264] The setting icon 611C can be used to listen for user touch operations via the icon. In response to this operation, the electronic device 100 can display a user interface for basic general settings of the electronic device 200. The options for these basic general settings may include, but are not limited to: icon transparency settings, button transparency settings, combo mode, restore default settings, save, return to the game interface, etc.

[0265] The issue feedback icon 611D can be used to listen for user touch operations via the icon. In response to the operation, the electronic device 100 can display a user interface for the user to provide feedback on related issues.

[0266] In some other embodiments, the icons in the aforementioned function debugging bar 611 may also display corresponding text information, such as "Start Matching," "Show / Hide," "Settings," "Reset Key Matching," etc. In other embodiments, some or all of the icon controls in the aforementioned function debugging bar 611 may not be displayed on the touchscreen of the electronic device 100, but rather as physical buttons on the electronic device 200. The electronic device 100 may respond to a user's touch operation (e.g., pressing) on ​​the physical buttons on the electronic device 200 and display the corresponding function debugging user interface. It is understood that this application does not limit how the electronic device 100 is triggered to display the corresponding function debugging interface.

[0267] It is understood that the above implementation scenarios and function debugging user interface are only used as examples to illustrate this application and do not constitute a limitation on this application.

[0268] like Figure 11 As shown, in response to a touch operation (e.g., a click) on the settings icon 611C, the electronic device 100 can display a user interface 66. This user interface 66 can display a general settings window 661, which may include a button transparency setting item 662, a button sensitivity setting item 663, a combo mode setting item 664, a restore default settings control 665, a save control 666, and a return to the game interface control 667. Wherein:

[0269] Button transparency setting item 662 can be referenced above. Figure 8 The button transparency setting item 412 in the user interface 41 will not be described in detail here.

[0270] Button sensitivity setting item 663 can be found in the previous section. Figure 8 The button sensitivity setting item 413 in the user interface 41 will not be described in detail here.

[0271] For combo mode setting item 664, please refer to the previous section. Figure 8The combo mode setting entry 414 in the user interface 41 will not be described in detail here.

[0272] The restore default settings control 665 can listen to touch operations applied to the control. In response to the operation, the electronic device 100 can clear the data of each setting item adjusted by the user in the general settings window and restore the default settings of each setting item of the electronic device 100 to their original settings.

[0273] The save control 666 can listen to touch operations applied to the control, and in response to the operation, the electronic device 100 can save the data of each setting item adjusted by the user.

[0274] The return-to-game interface control 667 can listen to touch operations performed on the control, and in response to the operation, the electronic device 100 can display the user interface 60 of the game application.

[0275] Figure 12 A schematic diagram of an electronic device 100 recognizing virtual buttons in a game interface is shown in one embodiment.

[0276] like Figure 12 As shown, when the electronic device 100 begins to recognize the button area in the user interface in response to a touch action (e.g., a click) on the game recognition icon 611A, the user interface 62 can be displayed.

[0277] User interface 62 may include the aforementioned Figure 9 The user interface 60 shown displays interface elements (such as controls, icons, text content, etc.). After the electronic device 100 recognizes the aforementioned directional button area and skill button area, the user interface 62 displays the directional button area and skill button area marked as highlighted, and may also display prompt information 621, re-identify icons 622, and establish game controller button mapping icons 623. Wherein:

[0278] The prompt message 621 is used to notify the user that the key area recognition has been completed, and can be the text message "Automatic recognition of game keys completed". In other embodiments, the prompt message 621 can also be a voice message or an icon. This application does not limit this.

[0279] The re-identification icon 622 can be used to listen for touch operations (such as clicks) performed on the icon. In response to the operation, the electronic device 100 re-identifies the directional button area and skill button area in the user interface.

[0280] The game controller button mapping icon 623 can be used to listen for touch operations (such as clicks) applied to the icon. In response to the operation, the electronic device 100 can establish a button mapping relationship with the electronic device 200.

[0281] It is understandable that the prompts 621, re-identify icons 622, and establish game controller button mapping icons 623 are independent of the game application's user interface. That is to say, the above prompts and icons are not related to the game application's user interface, and the above prompts and icons do not change when the game application's user interface changes.

[0282] Figure 13 A schematic diagram of the interface is shown in one embodiment, illustrating the process of establishing a key mapping between electronic device 100 and electronic device 200.

[0283] like Figure 13 As shown, in response to a touch action (e.g., a click) on the game controller button mapping icon 623, when the button mapping between electronic device 100 and electronic device 200 begins to be established, electronic device 100 can display user interface 63. When a mapping relationship is established between the virtual game buttons and the physical buttons of the game controller, a cursor icon 631 that can be manipulated by the user is displayed in user interface 63. The cursor icon 631 can be used to move over the identified button area. When a virtual button needs to be adapted, the user moves the cursor over the specified virtual button icon, causing the cursor to hover over the virtual button icon.

[0284] For example, such as Figure 13 As shown, the cursor icon 631 hovers over the highlighted basic attack skill button 625C. In response to a touch operation (e.g., pressing) on ​​the function button 201E on the electronic device 200, the electronic device 200 sends a first signal containing the key value of the function button 201E to the electronic device 100. Upon receiving the first signal containing the key value of the function button 201E, the electronic device 100 establishes a mapping relationship between the basic attack skill button 625C and the function button 201E, and the basic attack skill button 625C is no longer highlighted. After completing the mapping between the basic attack skill button 625C and the function button 201E, the electronic device 100 displays the user interface 64. In this user interface 64, the cursor icon 631 moves to the next highlighted directional button area 624. Upon detecting a touch operation (e.g., pressing) on ​​the joystick button 201A by the electronic device 200, the electronic device 100 establishes a mapping relationship between the directional button area 624 and the joystick button 201A, and the directional button area 624 is no longer highlighted. Once all buttons have established a mapping relationship with the electronic device 200, the electronic device 100 displays a user interface 65 where the cursor icon 631 disappears and all buttons are no longer highlighted.

[0285] In some embodiments, the cursor icon 631 can move automatically to the highlighted button area, or it can be moved by the user to the highlighted button area specified by the user. This application does not limit this.

[0286] In other embodiments, the electronic device 100 can perform a validity check on the key mapping. That is, when a user is establishing a key mapping relationship, in response to a touch operation (e.g., pressing) on ​​a physical key on the electronic device 200, the electronic device 200 sends a first signal containing the physical key value to the electronic device 100. When the electronic device 100 receives the first signal containing the physical key value, it checks whether the physical key value has been mapped to other virtual key icons on the electronic device 100. If so, a prompt message is displayed to inform the user that the physical key has been mapped to other virtual key icons. The prompt message can be text information, such as "This key has been mapped, please select again," or it can be voice information; this application does not limit this.

[0287] Specifically, for example, Figure 13 As shown, the basic attack skill button 625C on electronic device 100 has been mapped to the function button 201E on electronic device 200. At this time, the cursor icon 631 moves to the direction button area 631 where the button mapping has not yet been performed. If a touch operation (e.g., pressing) is performed on the function button 201E on electronic device 200, electronic device 200 sends a first signal containing the key value of function button 201E to electronic device 100. When electronic device 100 receives the first signal containing the key value of function button 201E, it detects that the key value of function button 201E has been mapped to the basic attack skill button 625C, and displays the text prompt message "This button has been mapped, please select again" to prompt the user that the physical button has been mapped to the virtual button icon.

[0288] In other embodiments, the electronic device 100 can also perform a reasonableness detection on the key mapping. That is, when a user is establishing a key mapping relationship, in response to a touch operation (e.g., pressing) on ​​a physical key on the electronic device 200, the electronic device 200 sends a first signal containing the physical key value to the electronic device 100. When the electronic device 100 receives the first signal containing the physical key value, it checks whether the physical key value can establish a correct mapping relationship with the selected virtual key icon. If not, a prompt message is displayed to prompt the user to reselect. The prompt message can be text information, such as "This key cannot match the selected key icon, please reselect," or it can be voice information; this application does not limit this.

[0289] Specifically, for example, Figure 13 As shown, when the cursor icon 631 moves to the directional button area 631 that has not yet been mapped, if in response to a touch operation (e.g., pressing) on ​​the function button 201E on the electronic device 200, the electronic device 200 sends a first signal containing the key value of the function button 201E to the electronic device 100. When the electronic device 100 receives the first signal containing the key value of the function button 201E, it detects that the key value of the function button 201E cannot establish a correct mapping relationship with the directional button area 631, and displays the text prompt message "This button cannot be matched with the selected button icon, please select again" to prompt the user to reselect a physical button (e.g., joystick button 201A) that can be correctly mapped to the directional button area 631.

[0290] Figure 14A , Figure 14B This diagram illustrates the interface effect of a game controller controlling buttons in a game application after electronic devices 100 and 200 have successfully established button mapping.

[0291] Once a mapping relationship is established between the virtual button icons on electronic device 100 and the physical buttons on electronic device 200, in response to a user's touch operation (e.g., pressing) on ​​the physical buttons on electronic device 200, electronic device 100 can display a new user interface (also known as a second user interface). This user interface can be partially or completely different from the user interface displayed by electronic device 100 when the physical buttons on electronic device 200 are not touched. Examples include user interface refreshes caused by game application interface transitions, character movement within a game application interface, or changes in the game scene. Specifically, for instance... Figure 14A , Figure 14B An example of the technical effect interface described above in this application is shown.

[0292] like Figure 14A As shown, a mapping relationship is established between the function key 201E in the electronic device 200 and the "basic attack" button icon 625C on the electronic device 100. When the electronic device 100 detects a touch operation (e.g., pressing) of the user's finger 712 on the function key 201E on the electronic device 200, the electronic device 100 generates a touch event for the "basic attack" button icon 625C, and the electronic device 100 displays the user interface 71 of the game character icon 601 launching a basic attack skill.

[0293] like Figure 14BAs shown, a mapping relationship is established between the joystick button 201A in the electronic device 200 and the directional button area 624 on the electronic device 100. When the electronic device 100 detects a touch operation (e.g., pushing to the right) of the user's finger 732 on the joystick button 201A on the electronic device 200, the electronic device 100 generates a touch event on the right directional button 624C in the directional button area 624, and the electronic device 100 displays a user interface 73 showing the game character icon 601 moving to the right in the game scene.

[0294] It is understood that this embodiment Figure 14A , Figure 14B The technical effect interface shown is only used to explain the embodiments of this application and does not constitute a specific limitation on this application.

[0295] Based on the above exemplary embodiments and exemplary application scenarios, the steps for establishing a key mapping between electronic device 100 and electronic device 200 are described in detail below.

[0296] Figure 15 A flowchart of a key mapping method provided in this application is shown. Figure 15 As shown, this application uses an electronic device 100 being a mobile phone and an electronic device 200 being a game controller as an example to describe the method in detail. The method may include:

[0297] S101. The mobile phone and the game controller establish the first connection.

[0298] Specifically, mobile phones and game controllers can have one or more of the following: Bluetooth (BT) module and WLAN module. The Bluetooth (BT) module can provide one or more Bluetooth communication solutions, including classic Bluetooth (Bluetooth 2.1) or Bluetooth Low Energy (BLE). The WLAN module can provide one or more WLAN communication solutions, including Wi-Fi Direct, Wi-Fi LAN, or Wi-Fi SoftAP.

[0299] The mobile phone can use one or more wireless communication technologies, such as Bluetooth or WLAN, to establish an initial connection with the game controller.

[0300] S102, The phone displays the first user interface.

[0301] Specifically, the first user interface may include multiple game buttons (also known as game controls), and these multiple game buttons may include a first game button (also known as a first control). For example... Figure 9 The user interface 60 shown may include a basic attack skill button 603C.

[0302] S103, The phone detected the first user action.

[0303] Specifically, the phone detected the first user action requesting the establishment of game key mappings, such as... Figure 12 The user is shown performing a touch operation (e.g., clicking) on ​​the game identification icon 611A.

[0304] S104. The mobile phone scans the first user interface and identifies multiple game buttons in the first user interface through the first image processing algorithm.

[0305] like Figure 12 As shown. Specifically, the button areas in the game screen are separate from the game screen; that is, the button areas do not change with the game screen, and the color contrast between the button areas and the surrounding game screen is relatively large. Therefore, the mobile phone can identify the button areas in the game screen using a first image processing algorithm. Then, to improve the accuracy of button area recognition, the mobile phone can combine the obtained touch frequency of the user in the game application screen to correct the first image processing algorithm, thereby accurately and effectively finding the button areas in the game screen.

[0306] In some embodiments, the first image processing algorithm may be an edge detection algorithm. How the edge detection algorithm identifies game buttons will be explained later and will not be elaborated here.

[0307] The edge detection algorithm is just one example. This application does not impose any special restrictions on the first image processing algorithm. Other methods that can recognize game buttons are also acceptable.

[0308] S105. Generate a corresponding virtual button at the position of each game button, that is, generate a first virtual button at the position of the first game button.

[0309] Specifically, such as Figure 12 As shown, once the phone successfully recognizes the location of the game buttons in the user interface, a corresponding virtual button can be generated at the location of each game button.

[0310] S106. Display the first cursor at the first virtual button.

[0311] Specifically, such as Figure 13 As shown, the basic attack skill button 625C displays the first cursor 631. It is not limited to displaying the first cursor; the first virtual button can also display a highlighted state, a flashing state, etc., as long as it can be recognized by the user as a state to be selected.

[0312] S107: The game controller detected that the user pressed the first physical button.

[0313] Specifically, such as Figure 13As shown, the game controller detected the user pressing the first physical button function button 201E.

[0314] S108, The game controller sends the first signal that the first physical button has been pressed.

[0315] The first signal can carry the identifier of the first physical button.

[0316] S109. Successfully established the first mapping relationship between the first game button and the first physical button. Similarly, establish mapping relationships for the remaining buttons one by one.

[0317] Specifically, such as Figure 13 As shown, the mobile phone can obtain the position information of the basic attack skill button 625C where the cursor icon 631 is hovering, as well as the identifier or key value of the first physical button function button 201E, and establish a mapping relationship between the two. Similarly, a mapping relationship is also established between the other virtual buttons on the mobile phone and the physical buttons on the game controller.

[0318] After the mapping relationship between the first game button and the first physical button is established, the first cursor can be moved from the first game button to the second game button. This application does not restrict the order of cursor movement when matching buttons, and the cursor can be moved to any game button that has not yet been mapped.

[0319] After establishing a mapping relationship between the game buttons on the mobile phone interface and the physical buttons on the game controller, the mobile phone can generate and store a first mapping table, which records the mapping relationship between the game buttons on the mobile phone interface and the physical buttons on the game controller.

[0320] S110. After the mapping relationship between all game buttons and the physical buttons of the game controller is established, the phone returns to display the first user interface.

[0321] Specifically, once the mapping between all the game buttons on the phone and the physical buttons on the game controller is established, the phone returns to display the game interface.

[0322] S111: The game controller detects that the user has pressed the first physical button.

[0323] like Figure 14A As shown, the game controller detected the user pressing the first physical button function button 201E.

[0324] S112, The game controller sends the first signal to the mobile phone that the first physical button has been pressed.

[0325] S113. The mobile phone receives a signal that the first physical button has been pressed, and triggers the first game button to be clicked according to the first mapping relationship.

[0326] S114. The phone displays a second user interface, which is the user interface displayed when the game function corresponding to the first game button is triggered.

[0327] like Figure 14A As shown, when the first physical button function button 201E on the game controller is pressed, the mobile phone receives the signal that the first physical button function button 201E has been pressed, triggering the event that the basic attack function button 625C has been clicked, and displaying the user interface 71 showing the corresponding basic attack skill being triggered.

[0328] In some embodiments, the electronic device 100 may also set a first duration threshold. When the cursor hovers over a virtual button icon for a period exceeding the first duration threshold, the electronic device 100 identifies the virtual button icon as one that needs to be adapted and obtains the position information of the virtual button icon. Therefore, it is understood that this application does not limit how the electronic device 100 determines which virtual button icon in the game application screen needs to be adapted.

[0329] The following is combined Figure 16 and Figure 17 Taking edge detection algorithms as an example, this section explains the steps involved in an electronic device 100 recognizing button areas on a game application screen. Figure 16 As shown, the method steps may include:

[0330] S201, Electronic device 100 performs grayscale processing on the first image to obtain the second image.

[0331] Specifically, the first image is the image transformed from the first user interface, which is the game interface. The first image is a color image, and the second image is a grayscale image obtained by processing the first image into grayscale.

[0332] Understandably, the current mainstream standard image representation method is 24-bit mode, which is RGB value encoded with 24 bits per pixel (BPP). It uses three 8-bit unsigned integers (0 to 255) to represent the intensity of red, green, and blue. 24-bit mode is used for common color exchange in true color and image file formats such as the Joint Photographic Experts Group (JPEG) or Tag Image File Format (TIFF). It can generate 16 million color combinations, many of which are no longer accurately distinguishable to the human eye. That is, electronic devices first store red (R) (8 bits), then green (G) (8 bits), and finally blue (B) (8 bits), a total of 24 bits, with 256 gradients for each color, interleaved in the file in the form of RGBRGBRGB... JPEG is an international image compression standard. The JPEG image compression algorithm provides good compression performance while maintaining relatively good reconstruction quality, and is widely used in image and video processing. TIFF is a flexible bitmap format primarily used to store images, including photographs and artwork.

[0333] Grayscale images have only 8 bits of image depth, therefore requiring less computation in image processing than color images. Although some color levels are lost, the overall and local color and brightness level distribution characteristics of the entire image make the description of the second image (grayscale image) consistent with the description of the first image (color image).

[0334] Therefore, the electronic device 100 can perform grayscale processing on the first image. Based on the importance of R, G, and B, and other indicators, the three different components are weighted and averaged. Since the human eye is most sensitive to green and least sensitive to blue, weighting and averaging the RGB components according to Formula 1 yields a more reasonable grayscale image. The electronic device 100 can obtain the grayscale value of each pixel in the first image according to Formula 1, thereby obtaining the grayscale-processed second image.

[0335] Gray = 0.3R + 0.6G + 0.1B (Formula 1)

[0336] S202, Electronic device 100 performs Gaussian filtering on the second image.

[0337] Specifically, in the embodiments of this application, performing Gaussian filtering on the second image is to perform weighted averaging on the gray values ​​of the second image. That is to say, for the gray value of each pixel in the second image, the final gray value of the pixel after Gaussian filtering is obtained by weighted averaging of its own value and other gray values ​​in the neighborhood.

[0338] Therefore, it can be seen that Gaussian filtering can be divided into two steps: 1. Obtaining the Gaussian template (i.e., the weight template). 2. Performing a weighted average.

[0339] In the process of weighted averaging of pixels, the normal distribution is obviously a desirable weight allocation mode. Since the image is two-dimensional, a two-dimensional Gaussian function is required, as shown in Equation 2:

[0340]

[0341] When calculating the weights, you only need to take the "center point" as the origin and assign weights to the other points according to their positions on the normal curve to obtain a weighted average.

[0342] For example, assuming σ = 1.5, the fuzzy radius is 1, and the coordinates of the center point are (0,0), then the coordinates of the 8 nearest points are shown in Table 1:

[0343] Table 1

[0344] (-1,1) (0,1) (1,1) (-1,0) (0,0) (1,0) (-1,-1) (0,-1) (1,-1)

[0345] Using the x-coordinate as the x-coordinate and the y-coordinate as the y-coordinate in the above coordinate system, the weight matrix with a fuzzy radius of 1 is shown in Table 2.

[0346] Table 2

[0347] 0.0453542 0.0566406 0.0453542 0.0566406 0.0707355 0.0566406 0.0453542 0.0566406 0.0453542

[0348] The sum of the weights of these 9 points equals 0.4787147. Due to the characteristics of the weight template, if only the weighted average of these 9 points is calculated, their sum must equal 1. Therefore, the above 9 values ​​need to be divided by 0.4787147 respectively to obtain the final weight matrix, as shown in Table 3.

[0349] Table 3

[0350] 0.0947416 0.118318 0.0947416 0.118318 0.147761 0.118318 0.0947416 0.118318 0.0947416

[0351] Since the blur radius of the exemplary embodiment of this application is 1, when performing Gaussian filtering calculations on a pixel, only the weighted average of the surrounding 8 pixels needs to be taken. For example, the grayscale values ​​of these 9 pixels are shown in Table 4. The grayscale value range is 0-255. The pixel located at the center of Table 4 is the pixel that needs to be processed in this application.

[0352] Table 4

[0353] 14 15 16 24 25 26 34 35 36

[0354] The grayscale values ​​of the pixels in Table 4 above are multiplied by the weights at the corresponding positions in Table 3, as shown in Table 5:

[0355] Table 5

[0356] 14×0.0947416 15×0.118318 16×0.0947416 24×0.118318 25×0.147761 26×0.118318 34×0.0947416 35×0.118318 36×0.0947416

[0357] The final values ​​are shown in Table 6:

[0358] Table 6

[0359] 1.32638 1.77477 1.51587 2.83963 3.69403 3.07627 3.22121 4.14113 3.4107

[0360] Adding the nine values ​​in Table 6 above gives the Gaussian filtered value for the center point. The calculation process for the Gaussian filtered value of the center point is shown in the following formula:

[0361] 1.32638 + 1.77477 + 1.51587 + 2.83963 + 3.69403 + 3.07627 + 3.22121 + 4.14113 + 3.4107 = 24.99999

[0362] Since the sign representing the grayscale value is a positive integer, it can be approximated as 25.

[0363] As can be seen from the exemplary embodiments of this application described above, the electronic device 100 can obtain the second image by repeating the above process for each pixel in the first image.

[0364] In other embodiments, x and y in the two-dimensional Gaussian function can also have other values, that is, the final weight matrix can also be other values, and this application does not limit this.

[0365] In some other embodiments, the first image may not be grayscaled, and the electronic device 100 may directly perform Gaussian filtering on the first image. That is, the electronic device 100 may perform Gaussian filtering on the three RGB channels of the first image separately.

[0366] S203, Electronic device 100 acquires the user's touch frequency.

[0367] Specifically, taking the aforementioned game application as an example, the electronic device 100 can acquire the first number of touch operations (such as clicks) performed by the user on the screen interface of the electronic device 100 in the context of the game application scenario.

[0368] Specifically, the number of touch operations (e.g., clicks) performed by the user on the screen of the electronic device 100 for the game application scenario can be obtained by the game controller application. This game controller application can access the first count of touch operations (e.g., clicks) performed by the user on the screen of the electronic device 100 for the game application scenario, stored in the internal storage space of the electronic device 100. This first count can be historical data of the number of touch operations (e.g., clicks) performed by the user on the screen of the electronic device 100 for the game application scenario; that is, this first count can be the number of touch operations (e.g., clicks) performed by the user on the screen of the electronic device 100 for the game application scenario before the game application performs button recognition or button matching operations.

[0369] In some embodiments, electronic device 100 may acquire the user's touch frequency for an application requiring button matching when establishing a communication connection with electronic device 200. In other embodiments, electronic device 100 may acquire the user's touch frequency for an application requiring button matching after performing grayscale processing on the first image. That is, the step of electronic device 100 acquiring the user's touch frequency for an application requiring button matching only needs to be completed before electronic device 100 processes edge detection using a linear interpolation formula; the specific order of occurrence is not limited in this application.

[0370] S204. The electronic device 100 performs edge detection on the second image by combining the user touch frequency obtained above.

[0371] Specifically, the electronic device 100 uses an edge detection algorithm to perform edge detection on the second image.

[0372] In this embodiment, the edge detection operator of the edge detection algorithm can be the Sobel edge difference operator or other operators. The Sobel edge difference operator can detect edges based on the phenomenon that the weighted difference of gray levels between the upper and lower, left and right neighboring pixels of a pixel reaches an extreme value at the edge. The Sobel edge difference operator calculates the difference G in the horizontal direction. x Vertical difference G y This allows us to determine the gradient magnitude (also called gradient strength) G and direction θ of a pixel. As shown in Equations 3 and 4, G is the gradient strength (also called gradient value), θ represents the direction, and arctan is the arctangent function:

[0373]

[0374]

[0375] If the gradient intensity G is greater than or equal to a certain threshold, then the pixel is considered an edge point.

[0376] The following describes, by way of example, how to calculate gradient intensity G and direction θ.

[0377] The Sobel operators in the x and y directions are as follows:

[0378]

[0379] Among them, S x This represents the Sobel operator in the x-direction, used to detect edges in the y-direction; S y This represents the Sobel operator in the y-direction, used to detect edges in the x-direction (the edge direction is perpendicular to the gradient direction).

[0380] If there is a 3x3 window A in the image, and the first pixel to calculate the gradient is e, then after convolving with the Sobel operator, the gradient values ​​of the first pixel e in the x and y directions are as follows:

[0381]

[0382]

[0383] Where * represents the convolution symbol, and sum represents the summation of all elements in the matrix. Based on formulas 3 and 4, the gradient strength and direction of the first gradient vector at the first pixel e can be calculated.

[0384] After calculating gradients in an image, edges extracted solely based on gradient values ​​are still blurry. Therefore, it's necessary to refine the calculated gradient edges, meaning retaining the local maximum gradient value while suppressing all other gradient values ​​to zero. This part of the algorithm consists of two steps: 1. Compare the gradient intensity of the current pixel with multiple pixels along the positive and negative directions. 2. If the gradient intensity of the current pixel is the largest compared to several other pixels, then that pixel is retained as an edge point; otherwise, the pixel is suppressed. Typically, for more accurate calculations, linear interpolation is used between several adjacent pixels spanning the gradient direction to obtain the gradient intensity to be compared.

[0385] For example, such as Figure 17As shown, taking a 5x5 window as the neighborhood of the current first pixel point e, the gradient intensity T(e) of the current pixel point e is compared with the gradient intensities T(Q1), T(Q2), T(Q3), and T(Q4) of pixels Q1, Q2, Q3, and Q4 on the window where the positive and negative gradient lines intersect. Here, S1 and S2 are two points on the same line as Q1, S3 and S4 are two points on the same line as Q2, S5 and S6 are two points on the same line as Q3, and S7 and S8 are two points on the same line as Q4.

[0386] As shown in Equations 5, 6, and 7, the linear interpolation formula for calculating the gradient intensity T(Q1) of Q1 is:

[0387]

[0388]

[0389] T(Q1)=e (w×T(S1)+(1-w)×T(S2)) Formula 7

[0390] Here, distance(S1,S2) represents the distance between points S1 and S2, and the coefficient w can be calculated from the gradient direction. Simultaneously, considering user habits, a weight m is added to areas with high touch frequency, where n is the number of times the current point S is clicked per minute, serving as a correction value for the gradient transformation of that area. Similarly, the gradient strength of other points Q1, Q2, Q3, and Q4 is calculated.

[0391] If the gradient strength of the current pixel S is the largest compared to the gradient strengths of other points Q1, Q2, Q3, and Q4 in the same direction, its value is retained. Otherwise, the gradient of the current pixel S is suppressed, that is, the gradient of the current pixel S is set to 0.

[0392] After refining the gradient edges as described above, the remaining pixels can more accurately represent the actual edges in the image. However, some edge pixels still exist in the image due to noise and color variations. To address these stray responses, edge pixels must be filtered with weak gradient values, while retaining edge pixels with high gradient values. This can be achieved by selecting high and low thresholds. A histogram of gradient intensities for all pixels in the entire image is plotted. The high-order gradient intensities corresponding to 75% of the total histogram are selected as the high threshold (also called the first threshold), and the low-order gradient intensities corresponding to 25% of the total histogram are selected as the low threshold. If a pixel's gradient value is higher than the high threshold, the pixel is retained; if the pixel's gradient value is lower than the low threshold, the pixel is excluded.

[0393] Finally, all the identified edge pixels are connected in sequence to form the boundary of the button area on the first image.

[0394] In other embodiments, the values ​​of the high and low thresholds can also be selected by other methods, and this application does not limit this.

[0395] S205. Electronic device 100 acquires multiple frames of images and repeats the above steps for each frame of image.

[0396] Specifically, the electronic device 100 can acquire a continuous set of images on a timeline and repeatedly perform Gaussian filtering and edge detection on each frame in the set. In other embodiments, the electronic device 100 can also acquire a set of images at certain time intervals. It is understood that this application does not limit how the electronic device 100 acquires multiple frames of images.

[0397] S206. Electronic device 100 compares the results of multiple sets of images and obtains the edge of the button area.

[0398] Specifically, the electronic device 100 can acquire the output results of multiple sets of images after Gaussian filtering and edge detection, and compare the results. The electronic device 100 can acquire the duplicate recognition position after comparing the output results of multiple sets of images, and this duplicate recognition position is the button area that the electronic device 100 needs to acquire.

[0399] In the above embodiments, depending on the context, the term "when..." can be interpreted as meaning "if...", "after...", "in response to determining...", or "in response to detecting...". Similarly, depending on the context, the phrase "when determining..." or "if (the stated condition or event) is detected" can be interpreted as meaning "if determining...", "in response to determining...", "when (the stated condition or event) is detected", or "in response to detecting (the stated condition or event)".

[0400] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0401] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A key mapping method, characterized in that, The method includes: The first electronic device establishes a first connection with the second electronic device; The first electronic device displays a first user interface, which is the user interface of a first game program. The first user interface includes multiple controls, which are operation controls of the first game program, and the multiple controls include a first control. The first electronic device detected the first user's operation; In response to the first user operation, the first electronic device performs edge detection on the image of the first user interface by combining the historical click count of each pixel in the first user interface, and identifies the boundaries of multiple controls in the first user interface; wherein, the higher the historical click count of the first pixel of the first user interface, the higher the probability that the first pixel is identified as the boundary point of the multiple controls. The first electronic device generates corresponding virtual buttons at the positions of the plurality of controls; The first electronic device sets the display state of the first virtual button to a pending selection state; the first virtual button is the virtual button corresponding to the first control; The first electronic device receives a first signal sent by the second electronic device through the first connection, and responds to the first signal received when the first virtual button is in a selected state, and establishes a mapping relationship between the first physical button of the second electronic device and the first control; the first signal is generated by the second electronic device when the first physical button is pressed by the user; After the first physical button and the first control establish a mapping relationship, the first electronic device switches the selected state to the second control among the plurality of controls, so that the second control can establish a mapping relationship with the physical button of the second electronic device.

2. The method as described in claim 1, characterized in that, Also includes: After establishing the mapping relationship between the first physical button of the second electronic device and the first control, the first electronic device selects the second control among the plurality of controls; The first electronic device receives the second signal sent by the second electronic device through the first connection, and responds to the second signal received when the second control is in the selected state, and establishes a mapping relationship between the second physical button of the second electronic device and the second control; the second signal is generated by the second electronic device when the second physical button is pressed by the user.

3. The method as described in claim 1, characterized in that, The second electronic device is a game controller.

4. The method as described in claim 1, characterized in that, Also includes: After the mapping relationship between the multiple controls and the multiple physical buttons of the second electronic device is established, the first electronic device receives the first signal through the first connection and executes the function corresponding to the first control.

5. The method as described in claim 1, characterized in that, The first electronic device identifies the boundaries of multiple controls in the first user interface based on the historical click counts of each pixel in the first user interface using an edge detection method, including: The first image obtained by the first electronic device through grayscale processing of the first user interface; The first electronic device uses an edge operator to calculate the gradient vector of each pixel in the first image; The first electronic device uses linear interpolation to compare the gradient value of the first gradient vector of the first pixel with the gradient values ​​of other pixels in the same direction as the first gradient vector of the first pixel. If the gradient value of the first pixel is the largest in the stated direction, then the first electronic device retains the gradient value of the first pixel and sets the gradient values ​​of the other pixels to zero. The first electronic device sets a first threshold. If the gradient value of the first pixel is greater than the first threshold, the first pixel is retained. All retained pixels constitute the boundary of the control in the first user interface.

6. The method as described in claim 5, characterized in that, The gradient value of the first pixel is related to the gradient value T(S1) of pixel S1, the gradient value T(S2) of pixel S2, the number of times the current pixel is clicked by the user per minute in the historical click count, the distance between points S1 and S2 (distance(S1,S2), and the distance between points Q1 and S1 (distance(Q1,S1), where S1 and S2 are two pixels on the same straight line as Q1.

7. The method as described in claim 4, characterized in that, The first electronic device performs the function corresponding to the first control, specifically including: the first electronic device displays a second user interface, which is different from the first user interface.

8. The method as described in claim 1, characterized in that, The first control is in a selected state, specifically including one or more of the following: the first control is in a highlighted state, a cursor is displayed on the first control, or the first control is in a blinking state.

9. The method according to any one of claims 1-8, characterized in that, Also includes: After the first electronic device displays the first user interface, the first electronic device displays a first floating control within the first user interface; Specifically, the first electronic device detects a first user operation, including detecting a user operation applied to the first floating control.

10. A communication method applied to a communication system, the communication system comprising a first electronic device and a second electronic device, characterized in that, include: The first electronic device establishes a first connection with the second electronic device; The first electronic device displays a first user interface, which is the user interface of a first game program. The first user interface includes multiple controls, which are operation controls of the first game program, and the multiple controls include a first control. The first electronic device detected the first user's operation; In response to the first user operation, the first electronic device performs edge detection on the image of the first user interface by combining the historical click count of each pixel in the first user interface, and identifies the boundaries of multiple controls in the first user interface; wherein, the higher the historical click count of the first pixel of the first user interface, the higher the probability that the first pixel is identified as the boundary point of the multiple controls. The first electronic device generates corresponding virtual buttons at the positions of the plurality of controls; The first electronic device sets the display state of the first virtual button to a pending selection state; the first virtual button is the virtual button corresponding to the first control; The second electronic device detects that the first physical button has been pressed by the user and generates a first signal; The second electronic device sends the first signal to the first electronic device through the first connection; The first electronic device responds to the first signal received when the first virtual button is in the selected state and establishes a mapping relationship between the first physical button and the first control. After the first physical button and the first control establish a mapping relationship, the first electronic device switches the selected state to the second control among the plurality of controls, so that the second control can establish a mapping relationship with the physical button of the second electronic device.

11. The method as described in claim 10, characterized in that, Also includes: After establishing the mapping relationship between the first physical button and the first control, the first electronic device selects the second control among the plurality of controls; The second electronic device detects that the second physical button has been pressed by the user and generates a second signal; The second electronic device sends the second signal to the first electronic device through the first connection; The first electronic device responds to the second signal received when the second control is in the selected state, and establishes a mapping relationship between the second physical button and the second control.

12. The method as described in claim 10, characterized in that, The second electronic device is a game controller.

13. The method as described in claim 10, characterized in that, Also includes: After the mapping relationship between the multiple controls and the multiple physical buttons of the second electronic device is established, the first electronic device receives the first signal through the first connection and executes the function corresponding to the first control.

14. The method as described in claim 10, characterized in that, The first electronic device identifies the boundaries of multiple controls in the first user interface based on the historical click counts of each pixel in the first user interface using an edge detection method, including: The first image obtained by the first electronic device through grayscale processing of the first user interface; The first electronic device uses an edge operator to calculate the gradient vector of each pixel in the first image; The first electronic device uses linear interpolation to compare the gradient value of the first gradient vector of the first pixel with the gradient values ​​of other pixels in the same direction as the first gradient vector of the first pixel. If the gradient value of the first pixel is the largest in the stated direction, then the first electronic device retains the gradient value of the first pixel and sets the gradient values ​​of the other pixels to zero. The first electronic device sets a first threshold. If the gradient value of the first pixel is greater than the first threshold, the first pixel is retained. All retained pixels constitute the boundary of the control in the first user interface.

15. The method as described in claim 14, characterized in that, The gradient value of the first pixel is related to the gradient value T(S1) of pixel S1, the gradient value T(S2) of pixel S2, the number of times the current pixel is clicked by the user per minute in the historical click count, the distance between points S1 and S2 (distance(S1,S2), and the distance between points Q1 and S1 (distance(Q1,S1), where S1 and S2 are two pixels on the same straight line as Q1.

16. The method as described in claim 13, characterized in that, The first electronic device performs the function corresponding to the first control, specifically including: the first electronic device displays a second user interface, which is different from the first user interface.

17. The method as described in claim 10, characterized in that, The first control is in a selected state, specifically including one or more of the following: the first control is in a highlighted state, a cursor is displayed on the first control, or the first control is in a blinking state.

18. The method according to any one of claims 10-17, characterized in that, Also includes: After the first electronic device displays the first user interface, the first electronic device displays a first floating control within the first user interface; Specifically, the first electronic device detects a first user operation, including detecting a user operation applied to the first floating control.

19. An electronic device, characterized in that, It includes a communication device, a touch screen, a memory, and a processor coupled to the memory, wherein the memory stores executable instructions, wherein: The communication device is used to establish a first connection with the second electronic device; The touchscreen is used to display a first user interface, which is the user interface of a first game program. The first user interface includes multiple controls, which are operation controls of the first game program, and the multiple controls include a first control. The touchscreen is also used to detect the first user operation; The processor is used to perform edge detection on the image of the first user interface by combining the historical click counts of each pixel in the first user interface, and to identify the boundaries of multiple controls in the first user interface; wherein, the higher the historical click count of the first pixel in the first user interface, the higher the probability that the first pixel is identified as the boundary point of the multiple controls. The processor is further configured to generate corresponding virtual buttons at the positions of the plurality of controls, and set the display state of the first virtual button to a pending state; the first virtual button is the virtual button corresponding to the first control; The communication device is also configured to receive a first signal sent by the second electronic device through the first connection; The processor is further configured to, in response to the first signal received when the first virtual button is in a selected state, establish a mapping relationship between the first physical button of the second electronic device and the first control; the first signal is generated by the second electronic device when the first physical button is pressed by the user; The processor is further configured to, after establishing a mapping relationship between the first physical button and the first control, switch the selected state to the second control among the plurality of controls, for establishing a mapping relationship between the second control and the physical button of the second electronic device.

20. The electronic device as claimed in claim 19, characterized in that, The processor is further configured to, after establishing the mapping relationship between the first physical button of the second electronic device and the first control, select the second control among the plurality of controls; The communication device is also configured to receive a second signal sent by the second electronic device through the first connection; The processor is further configured to, in response to the second signal received when the second control is in a selected state, establish a mapping relationship between the second physical button of the second electronic device and the second control; the second signal is generated by the second electronic device when the second physical button is pressed by the user.

21. The electronic device as claimed in claim 19, characterized in that, The second electronic device is a game controller.

22. The electronic device as claimed in claim 19, characterized in that, The processor is further configured to, after the mapping relationship between the plurality of controls and the plurality of physical buttons of the second electronic device is established, execute the function corresponding to the first control when receiving the first signal through the first connection.

23. The electronic device as claimed in claim 19, characterized in that, The processor is specifically used for: The first image obtained by performing grayscale processing on the first user interface; An edge operator is used to calculate the gradient vector of each pixel in the first image; Using linear interpolation, the gradient value of the first gradient vector of the first pixel is compared with the gradient values ​​of other pixels in the same direction as the first gradient vector of the first pixel. If the gradient value of the first pixel is the largest in the stated direction, then the gradient value of the first pixel is retained, and the gradient values ​​of the other pixels are set to zero. A first threshold is set. If the gradient value of the first pixel is greater than the first threshold, the first pixel is retained. All retained pixels constitute the boundary of the control in the first user interface.

24. The electronic device as claimed in claim 23, characterized in that, The gradient value of the first pixel is related to the gradient value T(S1) of pixel S1, the gradient value T(S2) of pixel S2, the number of times the current pixel is clicked by the user per minute in the historical click count, the distance between points S1 and S2 (distance(S1,S2), and the distance between points Q1 and S1 (distance(Q1,S1), where S1 and S2 are two pixels on the same straight line as Q1.

25. The electronic device as claimed in claim 22, characterized in that, The processor is specifically used to display a second user interface, which is different from the first user interface.

26. The electronic device as claimed in any one of claims 19-25, characterized in that, The first control is in a selected state, specifically including one or more of the following: the first control is in a highlighted state, a cursor is displayed on the first control, or the first control is in a blinking state.

27. A computer-readable storage medium comprising instructions, characterized in that, When the instructions are executed on an electronic device, the electronic device causes the electronic device to perform the method as claimed in any one of claims 1 to 9 and 10 to 18.

Images