A method for constructing a semantic map and related apparatus

Abstract

The application discloses a semantic map construction method and related device, which can realize the construction of a planar map and the identification of semantic interest points in image data and the positions of the semantic interest points in the planar map through image data and inertial navigation data collected by an electronic device. The electronic device can mark the semantic interest points in the planar map and display collection guidance prompts to guide the electronic device to the uncollected area in the indoor to construct a complete planar map of the indoor. In this way, the planar map with the semantic interest points can be completely collected and generated under the premise that the planar map information is not acquired by the electronic device in advance.

Description

Technical Field

[0001] This application relates to the field of mapping, and more particularly to a method for constructing a map and related apparatus. Background Technology

[0002] With the widespread demand for location-based services (LBS), accurate positioning and navigation are among the key indicators of a good LBS. Accurate navigation systems rely on pre-collected base maps; for example, dead reckoning combined with base map matching algorithms depends on vector plane maps, and geomagnetic algorithms require magnetic field strength maps. Electronic devices can provide related services with the assistance of navigation base maps and accurate positioning, such as finding parking spaces and recording parking positions in indoor parking garages. Therefore, floor plans play a crucial role in this process.

[0003] Currently, typical floor plans only provide visualization, and in most cases, the collection and production of map data are incomplete. For example, typical floor plans do not accurately mark navigation-related semantic points of interest (POIs). POIs could include vehicle / pedestrian entrances / exits, parking spaces, fire extinguisher locations, etc. Traditional map data collection methods require a floor plan and manual recording of routes and related POIs. This leads to wasted costs and manpower for large underground parking garages, and in the absence of a floor plan, it is impossible to manually label POIs and other map data. Summary of the Invention

[0004] This application provides a method and related apparatus for constructing semantic maps, which realizes the construction of a floor plan through collected image data and inertial navigation data, and guides electronic devices to complete the collection of indoor map data.

[0005] In a first aspect, this application provides a method for constructing a semantic map, comprising: an electronic device acquiring image data of a building in real time via a camera and acquiring inertial navigation data in real time via an inertial sensor; the electronic device constructing a plan view of a first region based on the image data and the inertial navigation data, and identifying a second region within the building for which the electronic device has not constructed a plan view; the electronic device identifying points of interest (POIs) in the image data located in the first region; the electronic device marking the POIs in the first region on the plan view of the first region; and the electronic device displaying a acquisition guidance prompt, which instructs the electronic device to move to the second region and construct a plan view of the second region.

[0006] This application provides a method for constructing a semantic map, which can build a floor plan using image data and inertial navigation data collected by an electronic device, and identify semantic points of interest (POIs) in the image data and their locations on the floor plan. The electronic device can mark the POIs on the floor plan and display acquisition guidance prompts, directing the electronic device to uncollected areas indoors to construct a complete floor plan of the interior. Thus, even without prior acquisition of floor plan information, the electronic device can still completely collect and generate a floor plan with POIs.

[0007] In one possible implementation, when the electronic device constructs a planar map of the first region based on the image data and the inertial navigation data, the method further includes: the electronic device acquiring map data information of the first region, wherein the map data information includes one or more of magnetic field strength information and wireless signal strength information; the wireless signal strength information includes one or more of Wi-Fi signal reception strength information and Bluetooth signal reception strength information. This allows the electronic device to acquire more map data information during the construction of the semantic planar map.

[0008] In one possible implementation, before the electronic device acquires image data of the building in real time via a camera and inertial navigation data in real time via an inertial sensor, the method further includes: the electronic device receiving a first input; in response to the first input, the electronic device launching a map acquisition application and displaying the map acquisition application interface. In this way, the construction of a semantic planar map is initiated through the map acquisition application.

[0009] In one possible implementation, the map acquisition application interface includes layer controls; after the electronic device constructs a plan view of a first area based on the image data and the inertial navigation data, the method further includes: the electronic device receiving a second input from the user regarding the layer controls; in response to the second input, the electronic device displays a layer settings window, the layer settings window including one or more of a plan view layer option control, a POI layer option control, a road centerline layer control, and a data acquisition location layer control; wherein, the plan view layer option control is used to trigger the electronic device to turn on or off the display of the plan view generated by the electronic device on the map acquisition application interface, the POI layer option control is used to trigger the electronic device to turn on or off the display of POIs on the map acquisition application interface, the road centerline layer control is used to trigger the electronic device to turn on or off the display of road centerlines on the map acquisition application interface, and the data acquisition location layer is used to trigger the electronic device to turn on or off the display of the locations where the electronic device has acquired map data information within the building on the map acquisition application interface.

[0010] In one possible implementation, the image acquisition application interface includes an image capture screen control; the method further includes: the electronic device receiving a third input from the user regarding the image capture screen control; and in response to the third input, the electronic device displaying an image captured in real time by the camera. This allows users to conveniently view the images captured by the electronic device within the building.

[0011] In one possible implementation, the map data acquisition application interface includes a sensor control; the method further includes: the electronic device receiving a fourth input from a user to the sensor control; in response to the fourth input, the electronic device displaying a sensor settings window, the sensor settings window including one or more sensor data acquisition switches, the one or more sensor data acquisition switches including one or more of a Wi-Fi data acquisition switch, a geomagnetic data acquisition switch, and a Bluetooth data acquisition switch; wherein the Wi-Fi data acquisition switch is used to trigger the electronic device to turn on or off the acquisition of Wi-Fi data, the geomagnetic data acquisition switch is used to trigger the electronic device to turn on or off the acquisition of geomagnetic data, and the Bluetooth data acquisition switch is used to trigger the electronic device to turn on or off the acquisition of Bluetooth data.

[0012] In one possible implementation, the map acquisition application interface includes an acquisition guidance control; the electronic device displays acquisition guidance prompts, specifically including: the electronic device displays a plan view of the first area on the map acquisition application interface; the electronic device receives a fifth input from the user regarding the acquisition guidance control; in response to the fifth input, the electronic device displays the acquisition guidance prompts on the plan view of the first area.

[0013] In one possible implementation, the electronic device constructs a plan view of a first region based on the image data and the inertial navigation data. Specifically, the electronic device identifies the positions of vehicle racks, pillars, and the plan view boundary line of the first region within the building in the image data; based on the positions of the vehicle racks and pillars within the building, the electronic device determines the road boundary line and the road center line within the building; the electronic device extends the road center line and determines the direction area where the extended road center line does not intersect with the plan view boundary line of the first region as the second region.

[0014] In one possible implementation, the POI includes one or more of a vehicle entrance / exit, a pedestrian entrance / exit, a parking space, and a fire extinguisher location.

[0015] In a second aspect, this application provides an electronic device, including: one or more processors, one or more memories, a camera, and an inertial sensor; wherein the camera, the inertial sensor, and the one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program code, the computer program code including computer instructions, which, when the one or more processors execute the computer instructions, cause the electronic device to perform any of the possible methods described above.

[0016] Thirdly, embodiments of this application provide a computer storage medium including computer instructions, which, when executed on an electronic device, cause the electronic device to perform the method in any of the possible implementations of any of the above aspects.

[0017] Fourthly, embodiments of this application provide a computer program product that, when run on a computer, causes the computer to execute the method in any of the possible implementations of any of the above aspects. Attached Figure Description

[0018] Figure 1 A schematic diagram illustrating an indoor positioning and navigation method provided in an embodiment of this application;

[0019] Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0020] Figure 3 This is a schematic diagram illustrating a semantic map construction scenario provided in an embodiment of this application;

[0021] Figure 4 A schematic diagram of the semantic map construction method provided in this application embodiment;

[0022] Figures 5A-5H A set of interface schematic diagrams provided for embodiments of this application;

[0023] Figure 6 A flowchart illustrating a semantic map construction method provided in this application embodiment;

[0024] Figure 7 This application provides a schematic diagram of an algorithm for determining uncollected areas.

[0025] Figures 8A-8D A schematic diagram of a set of plan views provided for embodiments of this application;

[0026] Figure 9 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0027] Figure 10This is a schematic diagram of the software functional structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0028] The technical solutions in the embodiments of this application will be clearly and thoroughly described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; the word "and / or" in the text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0029] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0030] The following describes an indoor positioning and navigation method.

[0031] Figure 1 A schematic diagram of an indoor positioning and navigation method mentioned in this application is shown.

[0032] like Figure 1 As shown, the construction of this semantic map includes the following steps:

[0033] S101. Electronic equipment acquires building structure diagrams and floor plans of each floor.

[0034] S102. The electronic device determines multiple location points inside the building and collects scene images of the interior at each location point.

[0035] S103. The indoor scene image acquired by the electronic device is associated with the location point where the image was acquired. This location point information is the pre-stored on-site location information.

[0036] During subsequent indoor positioning and navigation, the electronic device can acquire real-time scene images of the user's location. After acquiring the real-time scene images, the electronic device can send the real-time scene image information to the server. The server will then compare and identify the received real-time scene image information with several pre-stored scene image information in the database.

[0037] However, the above-mentioned indoor positioning and navigation methods require prior acquisition of building structure diagrams and floor plans of each floor, and manual recording of the collected images and the location of the collected images is required. It is not possible to automatically mark the location of the building on the floor plan using electronic devices.

[0038] Therefore, this application provides a method for constructing a semantic map, which can construct a floor plan using image data and inertial navigation data collected by an electronic device, and identify semantic points of interest (POIs) in the image data and their locations on the floor plan. The electronic device can mark the POIs on the floor plan and display collection guidance prompts, directing the electronic device to uncollected areas within the room to construct a complete floor plan. In this way, even without prior acquisition of floor plan information, the electronic device can still completely collect and generate a floor plan with POIs.

[0039] Figure 2 A schematic diagram of the structure of the electronic device 100 is shown.

[0040] The following description uses electronic device 100 as an example to illustrate the embodiment. It should be understood that... Figure 2 The electronic device 100 shown is merely an example, and the electronic device 100 may have more than Figure 2 The more or fewer components shown can be combined into two or more components, or they can have different component configurations. The various components shown in the figure can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0041] Electronic device 100 may include: processor 110, external memory interface 120, internal memory 121, universal serial bus (USB) interface 130, charging management module 140, power management module 141, battery 142, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0042] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0043] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0044] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0045] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0046] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0047] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C buses. The processor 110 can couple to the touch sensor 180K, charger, flash, camera 193, etc., through different I2C bus interfaces. For example, the processor 110 can couple to the touch sensor 180K through the I2C interface, enabling the processor 110 and the touch sensor 180K to communicate through the I2C bus interface, thereby realizing the touch function of the electronic device 100.

[0048] The I2S interface can be used for audio communication. In some embodiments, the processor 110 may include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to enable communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the I2S interface to enable the function of answering phone calls through a Bluetooth headset.

[0049] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via the PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface, enabling the function of answering phone calls through a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0050] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 via the UART interface to implement Bluetooth functionality. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the UART interface to enable music playback through Bluetooth headphones.

[0051] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI) and a display serial interface (DSI). In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to enable the electronic device 100 to capture images. The processor 110 and the display screen 194 communicate via the DSI interface to enable the electronic device 100 to display images.

[0052] The GPIO interface can be configured via software. It can be configured as a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 to a camera 193, a display screen 194, a wireless communication module 160, an audio module 170, a sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0053] USB port 130 is a USB standard compliant interface, specifically a Mini USB port, Micro USB port, USB Type-C port, etc. USB port 130 can be used to connect a charger to charge electronic device 100, and can also be used for data transfer between electronic device 100 and peripheral devices. It can also be used to connect headphones for audio playback. This interface can also be used to connect other electronic devices, such as AR devices.

[0054] It is understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0055] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device via the power management module 141.

[0056] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, external memory, display screen 194, camera 193, and wireless communication module 160, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be located in the same device.

[0057] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0058] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0059] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0060] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio device (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 150 or other functional modules.

[0061] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0062] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0063] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0064] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0065] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0066] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0067] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0068] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 100 selects a frequency, the DSP can perform Fourier transforms on the frequency energy.

[0069] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0070] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0071] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0072] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0073] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0074] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0075] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or make hands-free calls through the speaker 170A.

[0076] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the electronic device 100 answers a telephone call or voice message, the receiver 170B can be brought close to the ear to listen to the voice.

[0077] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C. Electronic device 100 may have at least one microphone 170C. In some embodiments, electronic device 100 may have two microphones 170C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, electronic device 100 may also have three, four, or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0078] The 170D headphone jack is used to connect wired headphones. The 170D headphone jack can be a USB 130 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0079] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A can be disposed on display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may include at least two parallel plates with conductive material. When force is applied to pressure sensor 180A, the capacitance between the electrodes changes. Electronic device 100 determines the pressure intensity based on the change in capacitance. When a touch operation is applied to display screen 194, electronic device 100 detects the intensity of the touch operation based on pressure sensor 180A. Electronic device 100 can also calculate the touch position based on the detection signal from pressure sensor 180A. In some embodiments, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation commands. For example, when a touch operation with an intensity less than a first pressure threshold is applied to the SMS application icon, a command to view an SMS is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to the SMS application icon, a command to create a new SMS is executed.

[0080] The gyroscope sensor 180B can be used to determine the motion attitude of the electronic device 100. In some embodiments, the gyroscope sensor 180B can determine the angular velocity of the electronic device 100 about three axes (i.e., the x, y, and z axes). The gyroscope sensor 180B can be used for image stabilization. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the shake of the electronic device 100, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to counteract the shake of the electronic device 100 by moving in the opposite direction, thus achieving image stabilization. The gyroscope sensor 180B can also be used in navigation and motion-sensing game scenarios.

[0081] The barometric pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates altitude using the air pressure value measured by the barometric pressure sensor 180C to assist in positioning and navigation.

[0082] The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip cover. In some embodiments, when the electronic device 100 is a flip phone, the electronic device 100 can detect the opening and closing of the flip cover using the magnetic sensor 180D. Then, based on the detected opening and closing state of the cover or the flip cover, features such as automatic flip unlocking can be set.

[0083] The 180E accelerometer can detect the magnitude of acceleration of electronic device 100 in various directions (typically three axes). When electronic device 100 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the posture of electronic devices and applied to applications such as screen orientation switching and pedometers.

[0084] A distance sensor 180F is used to measure distance. Electronic device 100 can measure distance via infrared or laser. In some embodiments, during a shooting scene, electronic device 100 can utilize the distance sensor 180F to measure distance for rapid focusing.

[0085] The proximity sensor 180G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The LED may be an infrared LED. The electronic device 100 emits infrared light outward through the LED. The electronic device 100 uses the photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100. The electronic device 100 may use the proximity sensor 180G to detect when a user holds the electronic device 100 close to their ear for a call, so as to automatically turn off the screen to save power. The proximity sensor 180G can also be used in holster mode and pocket mode for automatic unlocking and locking of the screen.

[0086] The ambient light sensor 180L is used to sense the brightness of ambient light. The electronic device 100 can adaptively adjust the brightness of the display screen 194 based on the sensed ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also work with the proximity sensor 180G to detect whether the electronic device 100 is in a pocket to prevent accidental touches.

[0087] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can utilize the characteristics of the collected fingerprints to achieve fingerprint unlocking, accessing application locks, taking photos with fingerprints, answering calls with fingerprints, etc.

[0088] Temperature sensor 180J is used to detect temperature. In some embodiments, electronic device 100 uses the temperature detected by temperature sensor 180J to execute a temperature handling strategy. For example, when the temperature reported by temperature sensor 180J exceeds a threshold, electronic device 100 performs thermal protection by reducing the performance of a processor located near temperature sensor 180J to reduce power consumption. In other embodiments, when the temperature is below another threshold, electronic device 100 heats battery 142 to prevent abnormal shutdown of electronic device 100 due to low temperature. In still other embodiments, when the temperature is below yet another threshold, electronic device 100 boosts the output voltage of battery 142 to prevent abnormal shutdown due to low temperature.

[0089] Touch sensor 180K, also known as a "touch panel," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touch screen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of electronic device 100, in a different position than display screen 194.

[0090] The bone conduction sensor 180M can acquire vibration signals. In some embodiments, the bone conduction sensor 180M can acquire vibration signals from the vibrating bone segments of the human vocal cords. The bone conduction sensor 180M can also contact the human pulse to receive blood pressure signals. In some embodiments, the bone conduction sensor 180M can also be incorporated into headphones to form bone conduction headphones. The audio module 170 can parse the voice signals from the vibrating bone segments of the vocal cords acquired by the bone conduction sensor 180M to realize voice functionality. The application processor can parse heart rate information from the blood pressure signals acquired by the bone conduction sensor 180M to realize heart rate detection functionality.

[0091] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. Electronic device 100 can receive button input and generate key signal inputs related to user settings and function control of electronic device 100.

[0092] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can correspond to touch operations performed on different applications (such as taking photos, playing audio, etc.). Motor 191 can also correspond to different vibration feedback effects for touch operations performed on different areas of the display screen 194. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0093] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0094] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to make contact with and separate from the electronic device 100. The electronic device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 simultaneously. The multiple cards can be of the same or different types. The SIM card interface 195 is also compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to realize functions such as calls and data communication. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0095] The following describes a semantic map construction scenario provided in the embodiments of this application.

[0096] Figure 3 This illustration shows a semantic map construction scenario provided in an embodiment of this application.

[0097] like Figure 3 As shown, the electronic device 100 can be placed on a vehicle. The vehicle carries the electronic device 100 and moves within the garage. The vehicle can be manually driven or automatically driven. When the vehicle is manually driven, the electronic device 100 can display data collection guidance information on a screen, instructing the user to maneuver the vehicle towards an uncollected area within the garage. When the vehicle is automatically driven, the electronic device 100 can establish a connection with the vehicle's automatic driving system. After identifying an uncollected area, the electronic device 100 can send a data collection guidance command to the automatic driving system. Upon receiving the command, the system can control the vehicle to drive towards the uncollected area within the garage.

[0098] In one possible implementation, the electronic device 100 can be a vehicle-to-everything (V2X) interaction device. This electronic device 100 can construct a floor plan of the garage using image data captured by a camera and inertial navigation data obtained by an inertial sensor. It can also identify semantic points of interest (IOIs) and their locations in the image data and mark these IIOs on the floor plan. After identifying uncollected areas within the garage, the electronic device 100 can control the vehicle to move towards these uncollected areas to construct a floor plan of the uncollected areas and collect map data information for those areas.

[0099] In this embodiment, the electronic device 100 can also be a mobile platform such as an unmanned aerial vehicle or a ground robot. The electronic device 100 can move to an uncollected area based on the generated data collection guidance information. The electronic device 100 can also be a wearable device such as AR / VR glasses. The electronic device 100 displays data collection guidance prompts on its screen to guide the user to move to an uncollected area.

[0100] The following is in conjunction with the above. Figure 3 The present invention describes a semantic map construction method provided in the embodiments of this application, based on the semantic map construction scenario shown.

[0101] Figure 4 This illustration shows a flowchart of the semantic map construction method in the above-described semantic map construction scenario in an embodiment of this application.

[0102] like Figure 4 As shown, the method for constructing this semantic map may include the following steps:

[0103] S401, Electronic device 100 is placed on the vehicle.

[0104] S402, The vehicle carrying electronic equipment 100 drove into the garage.

[0105] S403, Electronic device 100 starts the map data acquisition application.

[0106] S404. After the map data acquisition application is enabled, the electronic device 100 acquires inertial navigation data and image data inside the garage.

[0107] The inertial navigation data may include acceleration data and gyroscope data. The electronic device 100 can acquire acceleration data through an accelerometer and gyroscope data through a gyroscope.

[0108] S405, Electronic device 100 can create a regional plan using inertial navigation data and image data inside the garage.

[0109] S406, Electronic device 100 generates a navigation base map based on multiple regional floor plans.

[0110] S407, Electronic Equipment 100 collects map data and displays collection guidance prompts, guiding vehicles to move to uncollected areas within the garage.

[0111] The map data information may include any one of the following: semantic points of interest, magnetic field strength information, and wireless signal strength information.

[0112] Electronic device 100 can identify semantic targets from indoor image data acquired by a camera, and determine the location of the semantic target on a regional plan map by combining inertial navigation data and image data. Electronic device 100 can collect magnetic field strength information at its location on the regional plan map using a magnetometer. Electronic device 100 can collect wireless signal strength information at its location on the regional plan map using a wireless communication module. For example, electronic device 100 can collect Wi-Fi signal strength information at its location on the regional plan map using a Wi-Fi module. As another example, electronic device 100 can collect Bluetooth signal strength information at its location on the regional plan map using a Bluetooth module. The above examples are merely for explaining this application and should not be construed as limiting it.

[0113] The electronic device 100 can determine the uncollected areas within the garage based on image data and inertial navigation data, and output collection guidance prompts to direct the electronic device 100 to move towards the uncollected areas. How the electronic device 100 determines the uncollected areas can be found in subsequent embodiments.

[0114] S408 and electronic equipment 100 have collected map data of all collectable areas within the garage.

[0115] The following describes the map data acquisition application interface displayed by the electronic device 100 when constructing a semantic map in the embodiments of this application.

[0116] like Figure 5A As shown, the electronic device 100 can display a desktop 510. The desktop 510 may include a page displaying application icons, including multiple application icons (e.g., browser, stock, calculator, voice assistant, video, weather, theme, settings, map, gallery, memo, image capture, etc.). Optionally, a page indicator is displayed below the page displaying application icons to indicate the total number of pages on the desktop and the positional relationship between the currently displayed page and other pages. Optionally, a status bar is displayed above the page displaying application icons, which may include information such as a signal strength indicator, battery level, and time. Further optionally, a tray icon area may be present below the page indicator, which may include one or more tray icons (e.g., dialer, messaging, contacts, camera, etc.), and these tray icons may remain displayed during page switching.

[0117] Electronic device 100 can receive input (e.g., a click) from a user's image data acquisition application icon 511, and in response to this input, electronic device 100 can display, for example... Figure 5B The image data acquisition application interface 520 is shown. After the image data acquisition application is started, the electronic device 100 acquires inertial navigation data and image data within the garage. The electronic device 100 can then create a regional plan view using the inertial navigation data and the image data within the garage.

[0118] like Figure 5B As shown, the map acquisition application interface 520 may include a regional plan view 526 and one or more functional controls (e.g., layer control 521, my location control 522, acquisition screen control 523, sensor control 524, and acquisition guidance control 525, etc.). The layer control 521 can be used to trigger the electronic device 100 to select a layer to display on the map acquisition application interface 520 (e.g., plan view layer, POI layer, road centerline layer, acquisition location layer, etc.). The my location control 522 can be used to trigger the electronic device 100 to display its current location on the plan view. The acquisition screen control 523 can be used to trigger the electronic device 100 to display the image captured by the camera on the map acquisition application interface 520. The sensor control 524 can be used to trigger the electronic device 100 to select and enable the acquisition of signal strength (including one or more of Wi-Fi signal strength, geomagnetic signal strength, and Bluetooth signal strength, etc.). The data acquisition guide 525 control can be used to respond to a fifth user input, triggering the electronic device 100 to turn on / off the display of the data acquisition guide prompt on the map data acquisition application interface 520. This guide prompt can be used to instruct the user to operate the electronic device 100 to proceed to an unacquired area for data acquisition. Optionally, the map data acquisition application interface 520 may also include a map scale bar and map semantic annotation information.

[0119] Electronic device 100 can receive a third input (e.g., a click) from the user to the capture screen control 523, and respond to the third input, such as Figure 5C As shown, the electronic device 100 can display the real-time captured image 527 from the camera on the image acquisition application interface 520. When the electronic device 100 receives input from the user regarding the captured image control 523 again, the electronic device 100 can close the display of the real-time captured image 527 from the camera on the image acquisition application interface 520.

[0120] The electronic device 100 can receive user input (e.g., a click) to the acquisition guidance control 525, and respond to the input, such as... Figure 5DAs shown, the electronic device 100 can display a data acquisition guidance prompt 528 on the floor plan 526. This prompt 528 can be used to guide the electronic device 100 to collect map data information in indoor areas that have not yet been acquired. When the electronic device 100 receives input from the user regarding the data acquisition guidance control 525 again, the electronic device 100 can turn off the display of the data acquisition guidance prompt 528 on the map data acquisition application interface 520.

[0121] Optionally, the electronic device 100 may also display a capture guidance prompt 529 on the image 527 captured in real time by the camera.

[0122] Electronic device 100 can receive a fourth input (e.g., a click) from the user to the sensor control 524, and in response to the fourth input, such as Figure 5E As shown, the electronic device 100 can display a sensor settings window 530 on the map data acquisition application interface 520. This sensor settings window 530 may include one or more sensor data acquisition switches. For example, a Wi-Fi data acquisition switch 531, a geomagnetic data acquisition switch 532, and a Bluetooth data acquisition switch 533, etc. Specifically, the Wi-Fi data acquisition switch 531 can be used to trigger the electronic device 100 to turn on or off the acquisition of indoor Wi-Fi data (including Wi-Fi signal name, Wi-Fi signal strength, etc.). The geomagnetic data acquisition switch 532 can be used to trigger the electronic device 100 to turn on or off the acquisition of indoor geomagnetic data (including geomagnetic signal strength, etc.). The Bluetooth data acquisition switch 533 can be used to trigger the electronic device 100 to turn on or off the acquisition of indoor Bluetooth data (including Bluetooth name and Bluetooth signal strength).

[0123] like Figure 5F As shown, the electronic device 100 can receive a second input from the user to the layer control 521, and in response to the second input, such as Figure 5G As shown, the electronic device 100 can display a layer settings window 540 on the map data acquisition application interface 520. This layer settings window 540 may include controls 541 corresponding to the planar map layer options, controls 542 corresponding to the POI layer options, controls 543 corresponding to the road centerline layer, and controls 544 corresponding to the acquisition location layer. For example... Figure 5G As shown, controls 541, 542, and 544 are currently enabled, while control 543 is disabled. Electronic device 100 has enabled the display of the plan view layer, POI layer, and acquisition location layer, and disabled the display of the road centerline layer.

[0124] Electronic device 100 can receive user input (e.g., a click) to control 543, and in response to the input, electronic device 100 can enable the display of the road centerline layer.

[0125] like Figure 5H As shown, after the electronic device 100 can enable the display of the road centerline layer, the aforementioned control 543 can be switched to the enabled state, and the electronic device 100 can display the road centerline 551 on the plan view 526.

[0126] The above Figures 5A-5H The examples shown are for illustrative purposes only and should not be construed as limiting the scope of this application.

[0127] The following describes a method for constructing a semantic map provided in an embodiment of this application.

[0128] Figure 6 A flowchart illustrating a semantic map construction method according to an embodiment of this application is shown.

[0129] like Figure 6 As shown, the method may include the following steps:

[0130] S601, Electronic device 100 receives the first input.

[0131] The first input can be for the above Figure 5A Input (e.g., clicking) for the China Map Data Acquisition Application 511.

[0132] S602, In response to the first input, the electronic device 100 can display the map data acquisition application interface.

[0133] The image data acquisition application interface can be referenced from the above. Figures 5B-5H The textual description of the map data acquisition application interface 520 is omitted here.

[0134] S603, the electronic device 100 can determine whether the database contains an indoor navigation map of the building. If not, it will proceed to steps S604 to S610.

[0135] The electronic device 100 can first determine the geographical location of the building through global navigation satellite system (GNSS) positioning, base station positioning, or Wi-Fi-assisted positioning. Based on the building's geographical location, the electronic device 100 can search a database for a navigation map corresponding to that building's geographical location. The navigation maps in the database all have their geographical locations noted.

[0136] In one possible implementation, the navigation maps for buildings in the database are annotated with their geographical names. Electronic device 100 can determine the geographical name of the building using GNSS positioning, base station positioning, or Wi-Fi-assisted positioning. Then, electronic device 100 can search the data to find a navigation map corresponding to the building's geographical name.

[0137] In one possible implementation, the navigation maps for buildings in the database are annotated with their geographical names. Electronic device 100 receives the geographical name of the building from the user's input. Based on the geographical name of the building entered by the user, electronic device 100 can search the database to see if there is a navigation map corresponding to that geographical name.

[0138] In this embodiment of the application, the database can be stored on the electronic device 100 or on a server.

[0139] S604, Electronic device 100 acquires indoor image data through a camera and inertial navigation data through an inertial sensor.

[0140] The inertial navigation data may include acceleration data and gyroscope data. The inertial sensor may include an accelerometer and a gyroscope. The electronic device 100 can acquire acceleration data via an accelerometer and gyroscope data via a gyroscope.

[0141] S605, the electronic device 100 can determine the plan view of the area where the electronic device 100 is located and the current position of the electronic device 100 on the plan view based on image data and inertial navigation data, and record the map data information of the current position.

[0142] The map data may include one or more of the following: points of interest (POIs), magnetic field data, wireless signal data, etc. The electronic device 100 can mark the location and size of the POIs on the map. The POIs include one or more of the following: vehicle entrances / exits, pedestrian entrances / exits, parking spaces, and fire extinguisher locations.

[0143] Electronic device 100 can construct a semantic map based on image data and inertial navigation data, either through simultaneous localization and mapping (SLAM) or semantic visual simultaneous localization and mapping (Semantic-SLAM).

[0144] Specifically, after entering the indoor space, the electronic device 100 can identify semantic targets in the acquired image data. Then, the electronic device 100 can calculate the pixel distance the semantic target has moved between two frames of the image data. The electronic device 100 determines the geographical distance it has moved between the two frames using inertial navigation data. Based on the pixel distance the semantic target has moved between the two frames and the geographical distance it has moved, the electronic device 100 can determine the relative position of the semantic target to its current location. The semantic targets include one or more of the following: entrances / exits, parking spaces, pillars, road signs, fire extinguishers, etc.

[0145] In this embodiment, filter techniques can be used to optimize the semantic map construction results from image data and inertial navigation data. These filters include, but are not limited to, one or more of the following: Kalman filter, particle filter, graph optimization filter, etc. The electronic device 100 can use convolutional neural networks and deep learning techniques to identify the size and location of semantic targets in the image data through the aforementioned processing logic.

[0146] Electronic device 100 can perform dead reckoning (DR) positioning by using inertial navigation data acquired by inertial sensors, magnetic field strength acquired by magnetometers, and wireless signal strength (e.g., Wi-Fi or Bluetooth signals) acquired by wireless communication modules, to determine its current indoor location. Combining semantic targets with the relative position of its current indoor location, electronic device 100 constructs a plan view of the area where it is located.

[0147] Optionally, after determining its current location, the electronic device 100 can measure the magnetic field strength at its current location using a magnetometer.

[0148] Optionally, after determining its current location, the electronic device 100 can measure the received signal strength indication (RSSI) of the wireless signal through a wireless communication module. For example, the electronic device 100 can measure the Wi-Fi signal identifier and RSSI at its current location through a Wi-Fi communication module, and it can measure the Bluetooth signal identifier and RSSI at its current location through a Bluetooth communication module.

[0149] In some embodiments, when the database contains indoor navigation maps of buildings, the electronic device 100 can determine its current position on the indoor navigation map based on the indoor navigation map and semantic information from the currently acquired images. The electronic device 100 can display the indoor navigation map, showing its current position. Based on the acquired image data and inertial navigation data, the electronic device 100 can identify uncollected areas in the indoor navigation map and complete the acquisition of map data.

[0150] Optionally, when the database contains indoor navigation maps of buildings, the electronic device 100 can determine its current location on the indoor navigation map using magnetic field positioning or wireless signal positioning. This can improve the accuracy of positioning.

[0151] If the semantic information in the image currently acquired by the electronic device 100 does not match the semantic information in the indoor navigation map, the electronic device 100 can determine a plan view of the area where the electronic device 100 is located based on the acquired image data and inertial navigation data. The electronic device 100 can complete the acquisition of map data based on this plan view and use the plan view to correct areas in the indoor navigation map where the semantic information does not match.

[0152] S606, the electronic device 100 can determine whether the map data of all areas inside the building has been collected. If not, proceed to steps S607 to S609. If the map data of all areas inside the building has been collected, proceed to step S610.

[0153] Specifically, the algorithm flow for electronic equipment 100 to identify whether map data information for all areas of a building has been collected can be referred to as follows. Figure 7 The illustrated embodiment.

[0154] S607. When the map data of all areas inside the building has not been collected, the electronic device 100 determines the uncollected areas inside the building.

[0155] The algorithm for electronic device 100 to determine uncollected areas inside a building can be referenced as follows: Figure 7 The illustrated embodiment.

[0156] S608 and electronic device 100 can display data acquisition guidance prompts on the plan view.

[0157] The data collection guidance prompt is used to guide the electronic device 100 to move to an indoor area that has not been collected. This data collection guidance prompt can be as described above. Figure 5D The data collection guide shown is number 528.

[0158] S609 and electronic device 100 move to indoor areas that have not been sampled, based on the data collection guidance prompts.

[0159] The electronic device 100 can be mounted on a vehicle. After displaying data collection guidance prompts, the user can control the vehicle carrying the electronic device 100 to move within the building. The vehicle can be manually driven or automatically driven.

[0160] Optionally, when the vehicle is in autonomous driving mode, the electronic device 100 can establish a connection with the vehicle's autonomous driving system. After determining the uncollected area, the electronic device 100 can send a collection guidance instruction to the autonomous driving system. After receiving the collection guidance instruction, the autonomous driving system can control the vehicle to drive towards the uncollected area in the garage.

[0161] After the electronic device 100 moves to a new position, the electronic device 100 can continue to perform the above steps S604 to S606.

[0162] S610. When all map data information of the indoor area has been collected, the electronic device 100 can end the collection of map data information and save the collected map data information in the cloud or locally.

[0163] In this embodiment of the application, the uncollected area includes the second area where no plan view has been constructed, and the collected area includes the first area where a plan view has been constructed.

[0164] The semantic map construction method provided in this application does not require additional mapping sensors (such as LiDAR or ultrasound). It simply utilizes image data collected by a camera and inertial navigation data collected by an inertial sensor to construct a floor plan and identify semantic targets in the image data and their positions on the floor plan. The electronic device 100 can mark the semantic targets on the floor plan as semantic points of interest (POIs) and display acquisition guidance prompts to guide the electronic device to uncollected areas within the room to construct a complete floor plan. In this way, the electronic device 100 can generate a floor plan and collect map data without the need for specialized and expensive components. Furthermore, it can still generate a floor plan and complete the collection of map data even without prior acquisition of floor plan information.

[0165] The algorithm for determining the uncollected area by the electronic device 100 in the embodiments of this application is described below.

[0166] Figure 7 A schematic diagram of an algorithm for determining uncollected areas provided in an embodiment of this application is shown.

[0167] like Figure 7 As shown, the algorithm for determining uncollected areas includes the following steps:

[0168] S701, Electronic device 100 identifies the location of car racks, pillars, and indoor boundary lines in the image data on the floor plan.

[0169] After acquiring image data via a camera, the electronic device 100 can identify vehicle grids, pillars, and indoor boundary lines in the captured image data. Then, the electronic device 100 can calculate the pixel distance the vehicle grids and pillars have moved in two frames of the image data. The electronic device 100 uses inertial navigation data to determine the geographical distance it has moved between the two image frames. Based on the pixel distances the vehicle grids, pillars, and indoor boundary lines have moved in the two image frames and the geographical distance it has moved, the electronic device 100 can determine the relative positions of the vehicle grids, pillars, and indoor boundary lines to its current location.

[0170] The electronic device 100 can determine the positions of the parking space and the pillars on the plan based on the relative positions of the parking space, the pillars, and the indoor boundary line with the current position of the electronic device 100.

[0171] S702 and electronic equipment 100 determine the road boundary line and road center line based on the positions of the vehicle grid and pillars on the plan.

[0172] After determining the positions of the parking spaces and pillars, the electronic device 100 can determine two parallel road boundary lines inside the building along the parking space lines and the edges of the pillars.

[0173] The electronic device 100 can insert one or more road centerlines between two parallel road boundary lines. In this embodiment, two road centerlines are used for illustration.

[0174] S703, Electronic device 100 extends the road centerline and determines the intersection point of the extended road centerline with the indoor boundary line in the generated plan view, as well as the intersection point of the two indoor boundary lines in the generated plan view.

[0175] like Figure 8A As shown, after determining the positions of the indoor boundary line and the road center line, the electronic device 100 can extend the road center line. The extended road center line may intersect with the indoor boundary line in the generated plan view.

[0176] S704. Electronic device 100 can determine whether the indoor boundary lines in the generated floor plan are closed. If yes, proceed to step S705. If no, proceed to step S706.

[0177] S705. Electronic device 100 determines whether the extended road centerline in the generated plan has intersections with the indoor boundary line. If not, proceed to steps S706 to S707. If yes, proceed to step S708.

[0178] S706, Electronic Equipment 100 determines the uncollected area in the direction where the road centerline and the boundary line in the indoor plan do not intersect.

[0179] S707, Electronic device 100 displays a data acquisition guide prompt, guiding electronic device 100 to move to the unacquired area.

[0180] For example, such as Figure 8A As shown, the electronic device 100 has generated a plan view including two indoor boundary lines (indoor boundary line 811 and indoor boundary line 812) and two road boundary lines (road boundary line 821 and road boundary line 822). The electronic device 100 can interpolate two road center lines (road center line 831 and road center line 832) based on road boundary lines 821 and 822. Indoor boundary line 811 runs from south to north, and indoor boundary line 812 runs from west to east. Road center line 831 is used for vehicles traveling from east to west, and road center line 832 is used for vehicles traveling from west to east. The electronic device 100 is currently traveling from west to east on road center line 832. The intersection of indoor boundary lines 811 and 812 is at the southwest corner of the generated plan view. Road center line 831 intersects the north-south indoor boundary line 811 in the western direction of the generated plan view, but does not intersect with the indoor boundary line in the eastern direction of the generated plan view. The road centerline 832 intersects the indoor boundary line 811 in the west direction of the generated plan, but does not intersect it in the east direction. Therefore, the electronic device 100 can determine that the indoor boundary line in the generated plan is not closed, and that the uncollected area in the generated plan is east of the road centerline 832. The electronic device 100 can display a collection guidance prompt on the generated plan, guiding the electronic device 100 to continue moving eastward along the road centerline 832.

[0181] like Figure 8BAs shown, the electronic device 100 has generated a plan view including three indoor boundary lines (indoor boundary line 811, indoor boundary line 812, and indoor boundary line 813) and four road boundary lines (road boundary line 821, road boundary line 822, road boundary line 823, and road boundary line 824). The electronic device 100 can interpolate two road center lines (road center line 831 and road center line 832) based on road boundary lines 821 and 822, and interpolate two road center lines (road center line 833 and road center line 834) based on road boundary lines 823 and 824. The direction of indoor boundary line 811 is from south to north, the direction of indoor boundary line 812 is from west to east, and the direction of indoor boundary line 813 is from south to north. Road centerline 831 is used for vehicles traveling from east to west, road centerline 832 is used for vehicles traveling from west to east, road centerline 833 is used for vehicles traveling from north to south, and road centerline 834 is used for vehicles traveling from south to north. Electronic device 100 is currently traveling from west to east on road centerline 832 to road boundary 824. The intersection of indoor boundary lines 811 and 812 is at the southwest corner of the generated plan. The intersection of indoor boundary lines 812 and 813 is at the southeast corner of the generated plan. Road centerline 831 intersects indoor boundary line 811 in the western direction of the generated plan, but does not intersect with indoor boundary lines in the eastern direction of the generated plan. Road centerline 832 intersects indoor boundary line 811 in the western direction of the generated plan, and intersects indoor boundary line 813 in the eastern direction of the generated plan. The road centerline 834 intersects the indoor boundary line 812 in the south direction of the generated plan, but there is no intersection in the north direction. Since the electronic device 100 only collects map data from west to east along the road centerline 821, and not from east to west along the road centerline 832, the electronic device 100 can determine that the indoor boundary line in the generated plan is not closed, and that the uncollected area in the generated plan is north of road centerline 834 and west of road centerline 832. The electronic device 100 can display collection guidance prompts on the generated plan, instructing it to move north along road centerline 834 or west along road centerline 822.

[0182] like Figure 8C As shown, the electronic device 100 can continue to construct the floor plan and complete the collection of map data by following the collection guide prompts.

[0183] like Figure 8DAs shown, the generated floor plan includes four indoor boundary lines (indoor boundary line 811, indoor boundary line 812, indoor boundary line 813, and indoor boundary line 814). These indoor boundary lines 811, 812, 813, and 814 form a closed rectangle. All road centerlines in the generated floor plan intersect with the indoor boundary lines, and map data has been collected along the centerlines of both directions of each road. Therefore, the electronic device 100 can determine that the indoor floor plan has been successfully constructed.

[0184] S708, Electronic Equipment 100 completes the collection of map and data information.

[0185] The semantic map construction method provided in this application does not require additional mapping sensors (such as LiDAR or ultrasound). It simply utilizes image data collected by a camera and inertial navigation data collected by an inertial sensor to construct a floor plan and identify semantic targets in the image data and their positions on the floor plan. The electronic device 100 can mark the semantic targets on the floor plan as semantic points of interest (POIs). Based on an algorithm for determining uncollected areas, it outputs collection guidance prompts, directing the electronic device 100 to continue collecting map data in uncollected areas of the room to construct a complete floor plan. In this way, the electronic device 100 can generate a floor plan and collect map data without the need for specialized and expensive components. Furthermore, it can still generate a floor plan and complete the collection of map data even without prior acquisition of floor plan information.

[0186] The hardware structure of an electronic device 100 provided in another embodiment of this application is described below.

[0187] Figure 9 A schematic diagram of the hardware structure of an electronic device 100 provided in another embodiment of this application is shown.

[0188] It should be noted that, Figure 9 The electronic device 100 shown is merely an example, and the electronic device 100 may have more than Figure 9 The more or fewer components shown can be combined into two or more components, or they can have different component configurations. Figure 9 The various components shown can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0189] like Figure 9As shown, the electronic device 100 may include a processor 901, a memory 902, a wireless communication processing module 903, an antenna 904, a display screen 905, and an inertial sensor 906. The processor 901, memory 902, wireless communication processing module 903, antenna 904, display screen 905, inertial sensor 906, and camera 907 can be connected via a bus 908.

[0190] The processor 901 is used to read and execute computer-readable instructions. Specifically, the processor 901 mainly includes a controller, an arithmetic logic unit (ALU), and registers. The controller is primarily responsible for instruction decoding and issuing control signals for the operations corresponding to the instructions. The ALU is mainly responsible for storing register operands and intermediate operation results temporarily stored during instruction execution. Specifically, the hardware architecture of the processor 901 can be an Application-Specific Integrated Circuit (ASIC), MIPS, ARM, or NP architecture, etc.

[0191] The processor 901 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 201 is a cache memory. This memory can store instructions or data that the processor 901 has just used or that are used repeatedly. If the processor 901 needs to use the instruction or data again, it can directly retrieve it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 901, and thus improves the efficiency of the system.

[0192] Memory 902 is coupled to processor 901 and is used to store various software programs and / or multiple sets of instructions. Memory 902 can be used to store computer executable program code, which includes instructions. Processor 901 executes various functional applications and data processing of electronic device 100 by running the instructions stored in memory 902. Memory 902 may include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function, etc. The data storage area may store data created during the use of electronic device 100, etc. In addition, memory 902 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0193] The wireless communication module 903 can provide solutions for wireless communication applications on the audio device 200, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks) and Bluetooth (BT).

[0194] In some embodiments, the wireless communication module 903 may include a Bluetooth (BT) communication module 903A and a WLAN communication module 903B. One or more of the Bluetooth (BT) communication module 903A and the WLAN communication module 903B can listen to signals emitted by other devices, such as probe requests, scan signals, etc., and can send response signals, such as probe responses, scan responses, etc., enabling other devices to discover the electronic device 100 and establish wireless communication connections with other devices, communicating with other devices through one or more wireless communication technologies, including Bluetooth or WLAN. The Bluetooth (BT) communication module 903A can provide one or more Bluetooth communication solutions, including Classic Bluetooth (BR / EDR) or Bluetooth Low Energy (BLE). The WLAN communication module 903B may include one or more WLAN communication solutions, including Wi-Fi Direct, Wi-Fi LAN, or Wi-Fi SoftAP.

[0195] Antenna 904 is used to transmit and receive electromagnetic wave signals. Antennas from different communication modules can be reused or used independently to improve antenna utilization.

[0196] The display screen 905 is used to display images, videos, etc. In one possible implementation, the electronic device 100 may also include a touch sensor. The touch sensor is also called a "touch panel." The touch sensor can be disposed on the display screen 905, and the touch sensor and the display screen 905 together form a touch screen, also called a "touchscreen."

[0197] The inertial sensor 906 includes an accelerometer and a gyroscope. The inertial sensor 906 can be used to detect inertial navigation data of the electronic device 100.

[0198] Camera 907 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0199] The electronic device 100 can generate a planar map with semantic information based on the image data acquired in real time by the camera 907 and the inertial navigation data detected in real time by the inertial sensor 906.

[0200] The software functional structure of the electronic device 100 provided in the embodiments of this application is described below.

[0201] Figure 10 A schematic diagram of the software functional structure of an electronic device 100 provided in another embodiment of this application is shown.

[0202] It should be noted that, Figure 10 The electronic device 100 shown is merely an example, and the electronic device 100 may have more than Figure 10 The number of more or fewer functional modules shown.

[0203] like Figure 10 As shown, the electronic device 100 may include a positioning module 1001, a visual semantic recognition module 1002, and a mapping module 1003.

[0204] The positioning module 1001 can be used to locate the position of the electronic device 100 inside a building. The specific algorithm for locating the position of the electronic device 100 inside a building can be found in the aforementioned embodiments and will not be repeated here.

[0205] The visual semantic recognition module 1002 can be used to identify semantic targets such as pillars, parking spaces, road signs, and fire extinguishers in image data collected in real time by electronic device 100.

[0206] The mapping module can generate a planar map and mark POIs on the planar map based on the positioning results of the positioning module 1001 and the semantic recognition results of the visual semantic recognition module 1002. The algorithm flow for generating the planar map can be referred to the aforementioned embodiments and will not be repeated here.

[0207] The positioning module 1001, the visual semantic recognition module 1002, and the mapping module 1003 can also be used to execute the steps in the aforementioned method embodiments, which will not be described in detail here.

[0208] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for constructing a semantic map, characterized in that, include: Electronic devices acquire real-time image data of the building through cameras and real-time inertial navigation data through inertial sensors; The electronic device constructs a plan view of a first region based on the image data and the inertial navigation data, and determines a second region within the building where the electronic device has not constructed a plan view; The electronic device identifies semantic interest points (POIs) in the first region of the image data; The electronic device marks semantic interest points located in the first region in the plan view of the first region; The electronic device displays a data acquisition guide prompt, which instructs the electronic device to move to the second area to construct a plan view of the second area. Specifically, determining the second area within the building where the electronic device has not been included in the floor plan includes: The electronic device identifies the location of the car racks, pillars, and the boundary lines of the first area's plan view within the building in the image data; The electronic device determines the road boundary line and road center line within the building based on the positions of the vehicle compartment and the pillar within the building; The electronic device extends the road centerline and determines the direction area where the extended road centerline does not intersect with the boundary line of the first area's plan view as the second area.

2. The method according to claim 1, characterized in that, When the electronic device constructs a plan view of the first region based on the image data and the inertial navigation data, the method further includes: The electronic device collects map data information of the first area, wherein the map data information includes one or more of magnetic field strength information and wireless signal strength information; the wireless signal strength information includes one or more of Wi-Fi signal reception strength information and Bluetooth signal reception strength information.

3. The method according to claim 2, characterized in that, Before the electronic device acquires image data of the building in real time via a camera and acquires inertial navigation data in real time via an inertial sensor, the method further includes: The electronic device receives the first input; In response to the first input, the electronic device starts the map data acquisition application and displays the map data acquisition application interface.

4. The method according to claim 3, characterized in that, The map acquisition application interface includes layer controls; after the electronic device constructs a plan view of the first region based on the image data and the inertial navigation data, the method further includes: The electronic device receives a second input from the user for the layer control; In response to the second input, the electronic device displays a layer settings window, which includes one or more of a planar layer option control, a POI layer option control, a road centerline layer control, and a data acquisition location layer control. The planar layer option control triggers the electronic device to enable or disable the display of the planar map generated by the electronic device on the map data acquisition application interface. The POI layer option control triggers the electronic device to enable or disable the display of POIs on the map data acquisition application interface. The road centerline layer control triggers the electronic device to enable or disable the display of road centerlines on the map data acquisition application interface. The data acquisition location layer control triggers the electronic device to enable or disable the display of the locations within the building where the electronic device has acquired the map data information.

5. The method according to claim 3, characterized in that, The image acquisition application interface includes acquisition screen controls; the method further includes: The electronic device receives a third input from the user regarding the control on the captured screen; In response to the third input, the electronic device displays the image captured in real time by the camera.

6. The method according to claim 3, characterized in that, The map acquisition application interface includes sensor controls; the method further includes: The electronic device receives a fourth input from the user for the sensor control; In response to the fourth input, the electronic device displays a sensor settings window, which includes one or more sensor data acquisition switches, including one or more of a Wi-Fi data acquisition switch, a geomagnetic data acquisition switch, and a Bluetooth data acquisition switch. The Wi-Fi data acquisition switch is used to trigger the electronic device to turn on or off the acquisition of Wi-Fi data; the geomagnetic data acquisition switch is used to trigger the electronic device to turn on or off the acquisition of geomagnetic data; and the Bluetooth data acquisition switch is used to trigger the electronic device to turn on or off the acquisition of Bluetooth data.

7. The method according to claim 3, characterized in that, The map data acquisition application interface includes acquisition guidance controls; the electronic device displays acquisition guidance prompts, specifically including: The electronic device displays a plan view of the first area on the map data acquisition application interface; The electronic device receives a fifth input from the user regarding the data collection guidance control; In response to the fifth input, the electronic device displays the acquisition guidance prompt on a plan view of the first area.

8. The method according to any one of claims 1-7, characterized in that, The POI includes one or more of the following: vehicle entrance / exit, pedestrian entrance / exit, parking space, and fire extinguisher location.

9. An electronic device, characterized in that, include: One or more processors, one or more memories, a camera, and an inertial sensor; wherein the camera, the inertial sensor, and the one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program code, the computer program code including computer instructions, which, when the one or more processors execute the computer instructions, cause the electronic device to perform the method as described in any one of claims 1-8.

10. A computer storage medium, characterized in that, Includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method of any one of claims 1-8.

11. A computer program product, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method according to any one of claims 1-8.

Images