Code scanning method and electronic equipment

Abstract

The invention discloses a code scanning method and electronic equipment. The invention relates to the technical field of terminals. According to the method, the electronic equipment can judge whether an adjustment instruction needs to be issued or not for an original image. That is to say, the electronic device does not need to determine whether the camera parameters need to be adjusted through the application side, can determine whether the camera parameters need to be adjusted through the hardware abstraction layer, and then sends the adjustment instruction to the hardware layer through the hardware abstraction layer, thereby greatly shortening the duration of the application side for identifying the barcode.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a code scanning method and electronic device. Background Art

[0002] In barcode scanning scenarios, the electronic device may be unable to recognize the barcode due to the distance between the electronic device and the barcode, or the barcode itself is small, resulting in the barcode being included in the image captured by the electronic device. In this case, the electronic device may need to adjust the camera parameters to enable barcode recognition. How to complete barcode recognition faster is an urgent problem in this field. Summary of the Invention

[0003] The present application provides a method and electronic device for scanning barcodes. The electronic device no longer needs to determine whether the camera parameters need to be adjusted through the application side. It can determine whether the camera parameters need to be adjusted through the HAL layer, and then send adjustment instructions to the hardware layer through the HAL layer, which greatly shortens the time it takes for the application side to recognize the barcode.

[0004] In a first aspect, the present application provides a method for scanning a code, which is applied to an electronic device, wherein the electronic device includes: a code scanning application, a hardware abstraction layer HAL layer, and a hardware layer. The method includes: in response to starting the code scanning function of the code scanning application, using the first camera parameters to capture a first original image through the hardware layer, and the first original image includes a barcode; generating a first target image based on the first original image through the HAL layer, and in parallel executing the action of determining the first target camera parameters based on the first original image through the HAL layer; passing the first target camera parameters to the hardware layer through the HAL layer; capturing a second original image through the hardware layer using the first target camera parameters, and the second original image including a barcode; generating a second target image based on the second original image through the HAL layer; and obtaining the information carried by the barcode in the second target image through the code scanning application.

[0005] By implementing the method of the first aspect, the HAL layer can directly determine whether the camera parameters need to be adjusted based on the original image, without having to wait until application B fails to recognize the barcode in the target image before determining whether the camera parameters need to be adjusted. In this way, the hardware layer only needs to wait for a short period of time to complete the adjustment of the camera parameters, and then the barcode can be successfully recognized.

[0006] Specifically, the HAL layer determines whether camera parameters need to be adjusted for the original image in frame t and issues an adjustment instruction. At this point (i.e., when the adjustment instruction is issued), the application layer attempts to identify the barcode in the target image in frame t and fails to do so. After receiving the adjustment instruction, the hardware layer adjusts the camera parameters. Since adjusting camera parameters also takes some time, after completing the adjustment, the hardware layer can send the adjusted original image to the HAL layer. The HAL layer then sends the adjusted target image to application B. At this point (i.e., when application B receives the adjusted target image), it is already frame t+n. Here, n can be the ratio of the hardware layer adjustment time to the single-frame time. Therefore, if application B fails to recognize the barcode in frame t, it only needs to wait until it receives frame t+n to successfully recognize the barcode. This shortens the overall cycle time and improves the user scanning experience.

[0007] In combination with the first aspect, in some embodiments, after generating a first target image based on the first original image through the HAL layer and performing the action of determining first target camera parameters based on the first original image through the HAL layer in parallel, the method further includes: transmitting the first target image to the code scanning application through the HAL layer; identifying a barcode in the first target image through the code scanning application; and when the barcode in the first target image is identified through the code scanning application, adjusting the camera parameters of the electronic device to the first target camera parameters through the hardware layer. For example, reference Figure 5 The two steps S216a and S215 in .

[0008] In this way, the generation of the target image M1 based on the HAL layer and the determination of whether the camera parameters need to be adjusted are performed in parallel. S215 and S216a can also be performed in parallel. In other words, the hardware layer does not need to wait for application B to determine whether the camera parameters need to be adjusted before adjusting the camera parameters.

[0009] In combination with the first aspect, in some embodiments, passing the first target image to the code scanning application through the HAL layer specifically includes: in response to generating the first target image based on the first original image through the HAL layer, sending the first target image to the code scanning application through the HAL layer; passing the first target camera parameters to the hardware layer through the HAL layer specifically includes: in response to analyzing the first original image through the HAL layer, passing the first target camera parameters to the hardware layer through the HAL layer.

[0010] In combination with the first aspect, in some embodiments, the method also includes: obtaining second feature information of the barcode in the second original image through the HAL layer, wherein the action of obtaining the second feature information of the barcode in the second original image through the HAL layer and the action of generating a second target image based on the second original image through the HAL layer are performed in parallel; the information carried by the barcode in the second target image obtained through the code scanning application specifically includes: obtaining the information carried by the barcode in the second target image through the code scanning application based on the second feature information.

[0011] The second characteristic information includes information such as the location, type, and feature points of the barcode in the second target image. The HAL layer can obtain this information, including the location, type, and feature points of the barcode. Based on this information, the HAL layer can determine whether camera parameters need to be adjusted. Furthermore, the HAL layer can send this information to the corresponding application simultaneously with the original image, thereby reducing application processing time and improving processing efficiency.

[0012] In combination with the first aspect, in some embodiments, the second feature information includes a first feature point of the barcode in the second original image, and the first feature point is in a distorted state. The information carried by the barcode in the second target image obtained by the code scanning application based on the second feature information specifically includes: correcting the first feature point to obtain the corrected first feature point; and obtaining the information carried by the barcode in the second target image based on the corrected first feature point.

[0013] In this way, barcode recognition can be completed even in a small-angle scanning scenario.

[0014] In conjunction with the first aspect, in some embodiments, determining, by the HAL layer, first target camera parameters based on the first original image specifically includes: acquiring, by the HAL layer, first feature information of a barcode in the first original image; and determining, by the HAL layer, the first target camera parameters based on the first feature information of the barcode in the first original image. The first feature information includes information such as a position, category, and feature points of the barcode in the first target image.

[0015] In conjunction with the first aspect, in some embodiments, before obtaining the first feature information of the barcode in the first original image through the HAL layer, the method further includes: cropping the first original image through the HAL layer to obtain a first cropped image, where the first cropped image includes the barcode; obtaining the first feature information of the barcode in the first original image through the HAL layer specifically includes: obtaining the first feature information of the barcode in the first original image through the HAL layer based on the first original image and the first cropped image. Exemplarily, the cropped image can be obtained through a pre-screening module.

[0016] In this way, the proportion of the barcode in the image can be increased, and the success rate of the subsequent barcode information detection module in acquiring the characteristic information of the barcode can be improved.

[0017] In combination with the first aspect, in some embodiments, cropping the first original image through the HAL layer specifically includes: determining a first prior frame from a preset prior frame based on feature information of a barcode in a third original image through the HAL layer, and the third original image is collected before the first original image; cropping the area where the first prior frame is located in the first original image through the HAL layer to obtain the first cropped image.

[0018] The pre-screening module is deployed in the HAL layer. Based on the barcode position detection results in the previous frame, the HAL layer crops the area in the current frame where the barcode may be located. This produces a cropped image, increasing the barcode's image share, improving the barcode detection module's success rate in acquiring barcode feature information, and increasing the maximum recognition distance for barcode scanning.

[0019] In conjunction with the first aspect, in some embodiments, the first target camera parameter includes a first target zoom ratio, and determining the first target camera parameter based on the first original image through the HAL layer specifically includes: determining, through the HAL layer, a first proportion of a barcode in the second original image; and determining, through the HAL layer, the first target zoom ratio based on the first proportion, such that, when the position of the electronic device does not change, the proportion of the barcode in the original image captured using the first target zoom ratio is greater than the first proportion. For example, reference may be made to the following description of the zoom determination module.

[0020] In combination with the first aspect, in some embodiments, the first target camera parameter includes a first target exposure parameter, and determining the first target camera parameter based on the first original image through the HAL layer specifically includes: determining the first grayscale value of the entire or partial area where the barcode in the second original image is located through the HAL layer; determining the first target exposure parameter based on the first grayscale value through the HAL layer.

[0021] In conjunction with the first aspect, in some embodiments, the first target camera parameter includes a first target focus parameter, and determining the first target camera parameter based on the first original image through the HAL layer specifically includes: determining the first coordinate of the center of the barcode in the second original image through the HAL layer; and determining the first target focus parameter based on the first coordinate through the HAL layer. For example, reference may be made to the following regarding the 3A judgment module and Figure 11D Related introduction.

[0022] In combination with the first aspect, in some embodiments, the method further includes: using the first camera parameter to acquire the third original image through the hardware layer; generating a third target image based on the third original image through the HAL layer; and discarding the third target image through the code scanning application; wherein, acquiring the third original image is after acquiring the first original image and before acquiring the second original image. Exemplarily, the third original image may refer to Figure 5 The original image m2 shown in FIG, the third target image can refer to Figure 5 The target image M2 is shown in .

[0023] In combination with the first aspect, in some embodiments, the method further includes: discarding the first target image through the code scanning application. Figure 5 If the target image M2 is shown in , the electronic device discards the first target image through the code scanning application.

[0024] In a second aspect, the present application provides an electronic device comprising a memory, a processor, and computer instructions stored in the memory, wherein the processor executes the computer program to implement the method of the above-mentioned first aspect or any embodiment of the first aspect.

[0025] In a third aspect, the present application provides a chip system comprising a processing circuit and an interface circuit, wherein the interface circuit is used to receive code instructions and transmit them to the processing circuit, and the processing circuit is used to allow the code instructions to execute the method of the above-mentioned first aspect or any embodiment of the first aspect.

[0026] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program. When the computer program runs on a processor, it implements the method of the above-mentioned first aspect or any embodiment of the first aspect.

[0027] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when the computer program runs on a processor, implements the method of the above-mentioned first aspect or any embodiment of the first aspect.

[0028] The beneficial effects of the first aspect can be referred to for the second to fifth aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 An application scenario provided by an embodiment of the present application;

[0030] Figure 2 A timing diagram of the interactive process of scanning a code provided in an embodiment of the present application;

[0031] Figure 3A A schematic diagram of a process in which application A fails to parse a barcode according to an embodiment of the present application;

[0032] Figure 3B A schematic diagram of a process flow in which application A fails to detect the position of a barcode according to an embodiment of the present application;

[0033] Figure 4 A software framework diagram of the electronic device 100 provided in an embodiment of the present application;

[0034] Figure 5 A timing diagram of the interactive process of scanning a code provided in an embodiment of the present application;

[0035] Figures 6A-6C Schematic diagrams of several parallel executions provided in the embodiments of the present application;

[0036] Figure 7A The implementation provided by the embodiment of this application Figure 2 The timing diagram of the code scanning method shown;

[0037] Figure 7B The implementation provided by the embodiment of this application Figure 5 The timing diagram of the code scanning method shown;

[0038] Figure 8 Schematic diagram of a neural network model based on the CSP-PAN structure provided in an embodiment of the present application;

[0039] Figure 9A A schematic diagram of the QR Code structure provided in an embodiment of the present application;

[0040] Figure 9B A schematic diagram of correcting the characteristic points of an acquired barcode provided in an embodiment of the present application;

[0041] Figure 10 A schematic diagram of the process of obtaining characteristic points of a barcode provided in an embodiment of the present application;

[0042] Figure 11A A schematic diagram of setting a priori frame provided in an embodiment of the present application;

[0043] Figure 11B Schematic diagram of the possible barcode area in the original image of frame t-1;

[0044] Figure 11C Schematic diagram of input information and output information of the barcode information detection module provided in an embodiment of the present application;

[0045] Figure 11D A schematic diagram of the area where the barcode is located in the original image provided in an embodiment of the present application;

[0046] Figure 12 A flowchart of a code scanning method provided in an embodiment of the present application;

[0047] Figure 13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] Below in conjunction with the accompanying drawings in the embodiment of the present application, the technical scheme in the embodiment of the present application is described.Wherein, in the description of the embodiment of the present application, the terms used in the following embodiments are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification and appended claims of the present application, the singular expressions "a kind of", "said", "above", "the" and "this" are intended to also include, for example, "one or more" such expressions, unless there is a clear contrary indication in its context. It should also be understood that in the following embodiments of the present application, "at least one", "one or more" refer to one or more (including two). "At least one (item) of the following" or its similar expressions refer to any combination of these items, including any combination of single item (item) or plural items (item). For example, at least one (item) of a, b or c can represent: a, b, c, a and b, a and c, b and c, or a, b and c, wherein a, b, c can be single or multiple. The term "and / or" is used to describe a relationship between related objects, indicating that three possible relationships exist. For example, A and / or B can represent: A alone, A and B together, and B alone. A and B can be singular or plural. The character " / " generally indicates an "or" relationship between the related objects.

[0049] References to "one embodiment" or "some embodiments" etc. described in this specification mean that the specific features, structures or characteristics described in conjunction with the embodiment are included in one or more embodiments of the present application. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. appearing in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in another way. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in another way. The term "connected" includes direct and indirect connections, unless otherwise stated. "First" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated.

[0050] In the embodiments of this application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplarily" or "for example" is intended to present the relevant concepts in a concrete manner.

[0051] The term "user interface (UI)" in the following embodiments of this application refers to a medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface is a source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on an electronic device and finally presented as content that the user can recognize. The commonly used form of user interface is graphical user interface (GUI), which refers to a user interface related to computer operations that is displayed in a graphical manner. It can be a visual interface element such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc. displayed on the display screen of an electronic device.

[0052] First, some noun concepts involved in the embodiments of this application are introduced.

[0053] Barcodes are used to store information (referred to as barcode information), which can include one-dimensional barcodes (also known as one-dimensional codes) and two-dimensional barcodes (also known as two-dimensional codes). Not limited to the above types, barcodes can also include more types of barcodes, which are not limited to this.

[0054] A QR code is a form of coding that uses black and white images on a two-dimensional plane to record data information (referred to as QR code information) based on specific geometric images and rules.

[0055] Scanning a barcode can be understood as the process by which an electronic device captures an image containing a barcode, analyzes the barcode in the image, and obtains the barcode information.

[0056] Camera parameters may include one or more of the following: exposure parameters, color temperature, focus, focal length, etc. Exposure parameters may also include: aperture, shutter speed, sensitivity. The camera parameters of an electronic device affect the original image captured by the electronic device. Auto Exposure (AE), Auto White Balance (AWB), and Auto Focus (AF) can be collectively referred to as 3A. Among them, AE is used to adjust the exposure parameters of the camera, AWB is used to adjust the color temperature of the camera, and AF is used to adjust the focus position of the camera. Electronic devices can obtain better shooting effects by adjusting 3A.

[0057] Zooming refers to changing the focal length of a lens, thereby changing the angle of view and the size of the subject. Zooming can include optical zoom, digital zoom, hybrid zoom, and more. Zooming is different from focusing.

[0058] Figure 1 An application scenario provided by an embodiment of the present application is exemplified.

[0059] refer to Figure 1 , a user can use an electronic device to scan a code. For example, the electronic device first captures an image using a camera. The electronic device then obtains a barcode, such as a QR code, contained in the image. Finally, the electronic device parses the barcode to obtain the information contained in the barcode.

[0060] However, in some application scenarios, such as paying for parking or queuing to order food, the distance between the electronic device and the barcode is far, or the size of the barcode itself is small, and the barcode size contained in the image captured by the electronic device is small, resulting in the electronic device being unable to parse the barcode, affecting the user experience.

[0061] The following describes an interactive process for scanning a code provided in an embodiment of the present application.

[0062] Figure 2 An example of an interactive process sequence diagram for scanning a code provided in an embodiment of the present application is shown.

[0063] The electronic device includes application A, a hardware abstraction layer (HAL layer), and a hardware layer. Application A is an application installed in the electronic device with a code scanning function. Application A can be a system application, such as a camera application, or a third-party application, such as another code scanning application. In the embodiments of the present application, application A is not limited.

[0064] Figure 2 The interaction sequence diagram shown is mainly divided into three parts:

[0065] S101-S109: trigger to collect original images;

[0066] S110a-S110e: continuously generate target images;

[0067] S114a-S119: Apply A to recognize the barcode in the target image.

[0068] Figure 2 The interaction process sequence diagram shown specifically includes:

[0069] S101. Application A starts the code scanning function.

[0070] S102. Application A displays a code scanning interface.

[0071] S103-1. Application A calls the openCamera interface to send a request to the HAL layer to start the camera.

[0072] S103-2. The HAL layer sends a request to the hardware layer to start the camera.

[0073] S104. The hardware layer starts the camera.

[0074] S105: The hardware layer returns information indicating that the camera startup is complete.

[0075] S106. Application A starts the preview function.

[0076] S107. Application A calls the startPreview interface to send a request to the HAL layer to capture the original image.

[0077] S108. HAL layer configuration parameters and initialization processing.

[0078] S109. The HAL layer sends a request to the hardware layer to capture the original image.

[0079] In some implementations, after the hardware layer receives a request to capture the original image, the hardware layer may periodically and continuously capture the original image, and the HAL layer may also periodically generate the target image.

[0080] Exemplarily, after the hardware layer receives a request to capture the original image, step group SG11 ( S110 a - S113 a ) may be executed.

[0081] S110a. The hardware layer collects the original image m1.

[0082] S111a. The hardware layer sends the original image m1 to the HAL layer.

[0083] S112a. The HAL layer generates the target image M1.

[0084] S113a. The HAL layer sends the target image M1 to application A.

[0085] The electronic device can generate the target image M1 through the above step group SG11.

[0086] When the HAL layer sends the target image M1 to the application A, since the application A is in an idle state and can process the target image M1, the application A receives the target image M1.

[0087] After executing the above steps S110a-S113a, step group SG12 (S110b-S113b) is then executed.

[0088] S110b. The hardware layer collects the original image m2.

[0089] S111b. The hardware layer sends the original image m2 to the HAL layer.

[0090] S112b. The HAL layer generates the target image M2.

[0091] S113b. The HAL layer sends the target image M2 to application A.

[0092] Likewise, the electronic device may generate the target image M2 through the above step group SG12.

[0093] When the HAL layer sends the target image M2 to the application A, the application A is busy executing S114a and cannot process the target image M2. Therefore, the application A discards the target image M2.

[0094] Similarly, after executing the above steps S110b-S113b, step group SG13 (S110c-S113c), step group SG14 (S110d-S113d), and step group SG15 (S110e-S113e) are executed again. The specific steps included in step group SG13, step group SG14, and step SG15 can refer to the above step group SG11.

[0095] It is worth mentioning that Figure 2 Each of the above-mentioned step groups is used to generate one frame of the target image. That is, the execution duration of each step group is the single-frame duration set by the electronic device. In other words, each time the electronic device completes each step group, it can generate one frame of the target image. However, whether the application side receives the target image sent by the HAL layer depends on whether the application side is currently idle. If the application side is currently idle, the target image is received; if the application side is currently busy, the target image is discarded. For example, application A receives target images M1, M3, and M5, and discards target images M2 and M4.

[0096] The above step groups SG11 to SG15 are only for illustrative purposes and are not limited to Figure 2 The five frames of images shown in the figure can generate multiple frames of target images after the electronic device starts to capture the original images. In addition, the above step groups SG11 to SG15 are executed by the hardware layer and the HAL layer, and do not affect the execution of application A in the corresponding time period.

[0097] In some implementations, after receiving the target image, application A may process the target image and identify the barcode in the target image to obtain the barcode information.

[0098] Application A can recognize barcodes in the following ways:

[0099] First, apply A to obtain the position of the barcode in the target image.

[0100] If application A does not find any barcode in the target image, it determines that the recognition has failed, ends the current recognition process, and waits for the next frame of the target image to be reported.

[0101] If application A successfully finds the barcode in the target image, it will further analyze the barcode. The analysis of the barcode may include: correcting the barcode, removing noise, and obtaining the barcode information according to the barcode encoding rules.

[0102] If the barcode is successfully parsed, the barcode information can be obtained, which means the barcode is successfully recognized. Optionally, Application A can also jump directly to the interface corresponding to the barcode information. Optionally, after the barcode is successfully parsed, Application A can disable the barcode scanning function.

[0103] If parsing fails, recognition is considered a failure. Further analysis can be performed to determine whether zooming is necessary to improve the likelihood of successful barcode recognition. For example, if the barcode is too small, zooming is determined to be necessary. Application A then calls the callback function provided by the camera framework, issues a zoom command, terminates the recognition process, and waits for the next target image frame to be reported.

[0104] That is, successful barcode recognition includes: parsing the barcode and obtaining the barcode information. Failed barcode recognition includes: not finding the barcode, or failing to parse the barcode.

[0105] The following combination Figure 3A-Figure 3B Two situations where barcode recognition fails are described. For example, application A includes a position detection module and a parsing module.

[0106] Figure 3A The flowchart of the failure of application A to parse the barcode is shown as an example.

[0107] refer to Figure 3A, application A obtains the target image. Application A then inputs the target image into the position detection module to determine the position of the barcode in the target image. After determining the position of the barcode, application A crops the range where the barcode is located in the target image to obtain a cropped image. Application A then inputs the cropped image into the parsing module for parsing. After the parsing fails, application A issues a zoom instruction. Application A waits for the next frame of target image to repeat the process of barcode recognition. For example, application A obtains the target image M1 containing the QR code a1, determines the position of the QR code a1 in the target image M1 through the position detection module, and then parses the QR code a1, but the parsing fails.

[0108] In other words, the zoom instruction will not be issued until application A completes a barcode recognition process and determines that the parsing has failed. Only then can the hardware layer make corresponding adjustments based on the zoom instruction. However, most applications A with a barcode scanning function currently use various barcode recognition algorithms to implement a barcode recognition process, which usually takes tens to hundreds of milliseconds. In addition, during the process of application A recognizing the barcode, the HAL layer continues to send the target image to application A, but because application A is busy (for example, in the barcode recognition process), many target images sent by the HAL layer to application A will be discarded, such as the above-mentioned target image M2 and target image M4.

[0109] Figure 3B The flowchart of the failure of application A to detect the barcode position is shown as an example.

[0110] refer to Figure 3B , application A obtains the target image. Application A then inputs the target image into the position detection module to determine the position of the barcode in the target image. Application A fails to obtain the barcode position. When the proportion of the barcode in the target image is too small, it is more likely to fail to obtain the barcode position. Application A waits for the next frame of the target image to repeat the barcode recognition process. For example, application A obtains the target image M1 containing the QR code a2. Because the QR code a2 is too small, the position of the QR code a2 in the target image M1 cannot be obtained.

[0111] In other words, the zoom command can only be issued after the barcode's position is detected. If the barcode position detection fails, Application A will not issue the zoom command, and the hardware layer will not make any adjustments. If Application A fails to detect the barcode's position because the barcode's proportion in the target image is too small, then since the hardware layer will not perform zoom processing and will not be able to subsequently improve the barcode's proportion in the target image, it may continue to fail to recognize the barcode.

[0112] The following S114a to S117 may be the case where a zoom instruction is issued after the above-mentioned barcode parsing fails.

[0113] S114a. Apply A to recognize the barcode in the target image M1.

[0114] Illustratively, after executing S113a, application A obtains the target image M1 and executes S114a.

[0115] S115a. After the recognition fails, application A determines whether to zoom.

[0116] After executing S114a, application A can recognize the barcode in target image M1. Parsing the barcode in target image M1 fails. Application A can further determine whether zooming is required based on target image M1. If zooming is determined to be necessary, application A can issue a zoom instruction. Application A's issuance of the zoom instruction specifically includes S116-1 and S116-2.

[0117] S116-1. Application A calls the callback function to send a zoom instruction to the HAL.

[0118] S116-2. The HAL layer sends a zoom instruction to the hardware layer.

[0119] S117. Perform zoom processing at the hardware layer.

[0120] After receiving the zoom instruction, the hardware layer may execute S117. For example, Figure 2 As shown, the hardware layer completes S117 before executing S110e. That is, the hardware layer captures the original image m5 after zooming. The remaining original images m1, m2, m3, and m4 are all captured before zooming.

[0121] The following S114b to S115b may also be the case where a zoom instruction is issued after the above-mentioned barcode parsing fails.

[0122] S114b. Apply A to recognize the barcode in the target image M3.

[0123] Exemplarily, after executing S113a, application A obtains the target image M3 and executes S114a.

[0124] S115b. After the recognition fails, application A determines whether to zoom.

[0125] For example, S114a and S115a may refer to the above-mentioned S114a and S115a, which are not described in detail here.

[0126] The following S118 to S119 may be the case where the above analysis is successful and the barcode information is obtained.

[0127] S118. Apply A to recognize the barcode in the target image M5.

[0128] S119. Application A obtains the barcode information.

[0129] After obtaining the barcode information, application A can perform related operations. For example, the barcode information includes a link to a page, and the electronic device can jump to display the page indicated by the barcode information based on the barcode information.

[0130] from Figure 2 From the timing diagram shown, it can be seen that: if application A fails to recognize the barcode in the target image, it is necessary to determine whether the parameters need to be adjusted. The hardware layer needs to wait for a long time to complete the adjustment of the camera parameters before it can successfully recognize the barcode.

[0131] Specifically, after application A fails to recognize the barcode for the target image of frame t, it issues a zoom command. Since the recognition of the barcode takes a period of time, at this time (i.e., the moment when the zoom command is issued), the HAL layer has already obtained the target image of frame t+m. Among them, m can be the smallest integer greater than the ratio of the recognition time of application A to the single frame time. For example, the recognition time of application A can refer to Figure 2 The waiting time for the application side to recognize is shown in . After the hardware layer receives the zoom instruction, it performs the zoom process. Since the zoom process also needs to be executed for a period of time, after the zoom process is completed, the hardware layer can send the original image after zoom to the HAL layer, and the HAL layer will send the target image after zoom to application A. At this time (that is, the moment when application A receives the target image after zoom) it is already the t+m+nth frame. Among them, n can be taken as the smallest integer greater than the ratio of the hardware layer adjustment time to the single frame time. For example, the hardware layer adjustment time can refer to Figure 2 The waiting time for hardware layer adjustments is shown in . This means that if application A fails to recognize the barcode on the target image in frame t, it must wait until the image in frames t+m+n is acquired before it can successfully recognize the barcode. This results in a long waiting time and a poor user scanning experience.

[0132] For example, Figure 2 As shown, after the electronic device fails to recognize the barcode on the first frame of image, it may need to wait until the fifth frame of image is acquired before it can successfully recognize the barcode.

[0133] For example, assume that it takes 100ms for the user to turn on the code scanning function and trigger the camera, for example, executing S101 to S103-2. It takes 500ms for the electronic device to start the camera and start collecting the original image, for example, executing S104 to S109. The frame rate of the electronic device for collecting images is set to 30ms, for example, the execution time of step group SG1 is 30ms. The time it takes for the application to successfully identify the barcode is 40ms. The time it takes for the application to fail to identify the barcode is greater than 40ms. In this way, it may take 790ms from the time the user turns on the code scanning function to the time the electronic device successfully obtains the barcode information. In this way, the user has to wait for a long time and the scanning experience is poor.

[0134] To improve the above-mentioned problem, an embodiment of the present application provides a barcode scanning method. In this method, the electronic device can determine whether to issue an adjustment instruction based on the original image. In other words, the electronic device no longer needs to determine whether the camera parameters (such as zoom) need to be adjusted through the application side. It can determine whether the camera parameters need to be adjusted through the HAL layer, and then send the adjustment instruction to the hardware layer through the HAL layer, which greatly shortens the time it takes for the application side to recognize the barcode.

[0135] Furthermore, the HAL layer's step of determining whether camera parameters need to be adjusted does not affect the application's acquisition of the target image. In other words, the HAL layer can generate the target image and determine whether camera parameters need to be adjusted in parallel. This allows the application to display the acquired target image at a normal frame rate and recognize the QR code within the target image. The application can also shorten the time it takes to recognize the barcode within the target image, allowing the electronic device to acquire the barcode information more quickly, allowing the user to more quickly understand the barcode information and providing a better scanning experience.

[0136] Before introducing the code scanning method provided in the embodiment of the present application, the software structure of the electronic device involved in the embodiment of the present application is first introduced.

[0137] Figure 4 The software framework diagram of the electronic device 100 provided in an embodiment of the present application is exemplarily shown.

[0138] The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes a layered architecture system as an example to exemplify the software structure of the electronic device 100.

[0139] A layered architecture divides software into several layers, each with distinct roles and responsibilities. Layers communicate with each other via software interfaces. In some embodiments, the electronic device system is divided into four layers: the application layer, the system framework layer, the hardware abstraction layer, and the hardware layer, from top to bottom.

[0140] The application layer can include a series of application packages. Figure 4As shown, the application package may include applications such as a camera application, a code scanning application, and a gallery application.

[0141] The camera application may refer to a system application of the electronic device, an application for managing camera services.

[0142] A barcode scanning app refers to an app with barcode scanning functionality. This can be a system app or a third-party app, such as a social networking app or lifestyle app. A barcode scanning app can access the electronic device's camera service through the camera app, meaning the electronic device can use the barcode scanning app to identify barcodes in real-time captured images.

[0143] The gallery application stores images. Recognizing barcodes in real-time target images is not limited to other scenarios. Electronic devices can also use barcode scanning applications to recognize barcodes in stored images in other scenarios.

[0144] In the embodiments of the present application, the names of the above-mentioned various applications are merely illustrative and do not limit their functions.

[0145] The application layer can also provide a user interface for user interaction. Electronic devices can activate the barcode scanning function through user actions on the user interface, which in turn displays the barcode scanning interface. For example, the barcode scanning interface may include a preview interface, a flash control, and more. The application layer not only handles basic decoding processes but also integrates intelligent scene recognition, machine learning, and other functions. For example, the application layer may include a business logic module for barcode recognition.

[0146] In some implementations, the barcode scanning application is used to obtain the target image sent by the HAL layer, to recognize the barcode on the target image, and to obtain the barcode information from the target image.

[0147] The system framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The framework layer includes predefined functions. For example, the framework layer includes a camera framework, an image processing framework, and various APIs. The framework layer provides functionality and interfaces for upper-layer applications.

[0148] The Hardware Abstraction Layer (HAL) is a bridge between the hardware of an electronic device and the operating system of the electronic device. The HAL layer can encapsulate the drivers of sensors from different manufacturers and provide a consistent interface for the upper layer, so that the upper layer software does not have to worry about the implementation details of the underlying hardware, thereby improving the portability and reusability of the code. In the embodiments of the present application, the specific implementation of the HAL layer in the camera function is mainly introduced as an example. The HAL layer can specifically include: a camera HAL layer (also known as a camera HAL layer).

[0149] The HAL layer is used to generate a target image based on the original image. Specifically, after receiving the original image, the HAL layer converts the original image into the target image required by the application layer. For example, the HAL layer may receive the resolution and pixel format specified by the application layer and generate a target image with the specified resolution and pixel format based on the original image.

[0150] In the embodiment of the present application, the HAL layer is not limited to adjusting the resolution and pixel format. It can also perform more specific processing on the original image, such as denoising and contrast enhancement, etc., to obtain the target image, which is not limited to this.

[0151] The HAL layer is also used to send the target image to the application layer. For example, the HAL layer can define a standardized interface (such as ICameraDeviceSession) through HIDL or AIDL, encapsulate the target image as a GraphicBuffer object, and pass it to the application layer through ANativeWindowBuffer.

[0152] In the embodiment of the present application, the HAL layer includes: a pre-screening module, a barcode information detection module, and a camera behavior judgment module.

[0153] The barcode information detection module includes modules such as a barcode position detection module and a barcode feature point detection module. The barcode information detection module is used to determine the feature information of the barcode in the original image. The barcode feature information includes one or more of the following: the barcode's location, category, feature points, etc.

[0154] The pre-screening module is used to predict the area where the barcode may exist in the original image.

[0155] The camera behavior judgment module is used to determine whether the camera parameters need to be adjusted and whether the corresponding adjustment instructions need to be sent to the hardware layer. The camera behavior judgment module includes: a zoom judgment module, a 3A judgment module and other modules. Among them, the zoom judgment module is used to determine whether zooming is required and to determine the target zoom ratio after determining that zooming is required. The 3A judgment module is used to determine whether to adjust camera parameters such as exposure parameters, color temperature, and focus. Exposure parameters, color temperature, and focus can be called 3A parameters. After determining that the hardware layer needs to adjust the corresponding parameters, the camera behavior judgment module can send corresponding instructions (such as zoom instructions, 3A adjustment instructions, etc.) to the camera driver, so that the hardware layer can make corresponding adjustments, thereby improving image quality.

[0156] It's worth noting that, since most barcodes are black and white, the barcode size in the original image has the greatest impact on barcode recognition. In other words, the camera parameter that most influences barcode recognition is focal length. Based on this, the camera behavior determination module primarily determines whether zooming is necessary and the target zoom ratio. Optionally, other camera parameters may also have a certain impact on barcode recognition, and the camera behavior determination module can also determine whether other camera parameters need to be adjusted, but this is not limited to this.

[0157] For the above-mentioned pre-screening module, barcode information detection module, and camera behavior judgment module, please refer to the subsequent more detailed introduction, which will not be repeated here.

[0158] Optionally, a kernel layer is included between the HAL layer and the hardware layer of the electronic device. The kernel layer includes camera driver, display driver, audio driver, and sensor driver, etc.

[0159] The hardware layer includes a variety of hardware devices in electronic devices. The various hardware responsible for the camera system in the hardware layer can be called cameras. Camera hardware may include a camera module, an image processor, a GPU, an NPU, and the like. Among them, the camera module may include one or more lens modules, one or more image sensors, an image signal processor (ISP), and the like. Optionally, the camera module may also include a time of flight (TOF) sensor, a multispectral sensor, and the like. The lens module is the core component of the camera module and is responsible for focusing and imaging light. The lens module may include one or more of the following lenses: main camera, telephoto lens, macro lens, ultra-wide-angle lens, depth of field lens. Among them, the telephoto lens is used to shoot distant objects and can achieve optical zoom.

[0160] The hardware layer is used to capture raw images. Specifically, an object is projected through a lens into a photosensitive element, which can be a CCD or CMOS phototransistor. The photosensitive element converts the optical signal into an electrical signal, which is then transmitted to the ISP for conversion into a digital image signal, i.e., the raw image. For example, the raw image can be unprocessed image data in Bayer GRBG format. The hardware layer is also used to send the raw image to the HAL layer via a hardware interface. The hardware layer can also adjust the lens and focal length based on adjustment instructions issued by the HAL layer.

[0161] It is understandable that both the original image and the target image may include image data collected by the electronic device, and the functions, data formats, and processing methods of the two are different.

[0162] In the embodiments of this application, Figure 4The software framework shown is for illustrative purposes only. The software framework of the electronic device may also include other forms of software frameworks, which are not limited thereto.

[0163] Combined with the above Figure 4 With reference to the related description, the following introduces an internal interaction process of scanning a code provided in an embodiment of the present application.

[0164] Figure 5 An internal interaction process sequence diagram of scanning a code provided in an embodiment of the present application is exemplified.

[0165] The electronic device includes application B, hardware abstraction layer (HAL layer), and hardware layer. Application B is an application with a code scanning function installed in the electronic device. Application B can be a system application, such as a camera application, or a third-party application, such as other code scanning applications. In the embodiment of the present application, application B is not limited. For detailed descriptions of application B, HAL layer, and hardware layer, please refer to the above Figure 4 The relevant introduction of , will not be repeated here.

[0166] Figure 5 The interaction sequence diagram shown is mainly divided into three parts:

[0167] S201-S209: triggering the acquisition of original images;

[0168] S210a-S213c: continuously generating target images and continuously determining whether camera parameters need to be adjusted;

[0169] S216a-S216b: Apply B to recognize the barcode in the target image.

[0170] Figure 5 The interaction process sequence diagram shown specifically includes:

[0171] S201. Application B starts the code scanning function.

[0172] Typically, for privacy and security reasons, application B is prevented from unauthorized access and use of the electronic device's camera hardware. The electronic device can initiate the code scanning function through application B after receiving a user operation. This user operation can include one or more of the following: touch, click, gesture, knuckle, voice, and eye movement. For example, this user operation can refer to clicking the code scanning control provided in application B.

[0173] In some implementations, before executing S201, the electronic device may launch application B in response to a user operation, and then execute S201. The electronic device may also launch application B and the code scanning function simultaneously. For example, the electronic device may execute S201 in response to a control on the desktop for launching the code scanning function of application B. In the embodiments of the present application, the specific method for executing S201 is not limited.

[0174] S202. Application B displays the code scanning interface.

[0175] In some implementations, after application B starts the code scanning function, the electronic device can start a camera application. Application B can use the camera service provided by the electronic device through the camera application.

[0176] After executing the above S201, the electronic device may display the code scanning interface provided by application B. At this time, the electronic device may trigger the camera to start. Specifically, the electronic device triggering the camera to start may include S203-1 and S203-2.

[0177] S203-1. Application B calls the openCamera interface to send a request to the HAL layer to start the camera.

[0178] S203-2. The HAL layer sends a request to the hardware layer to start the camera.

[0179] S204. The hardware layer starts the camera.

[0180] In some implementations, the hardware layer may start the camera after receiving a request to start the camera sent by the HAL layer.

[0181] S205: The hardware layer returns information indicating that the camera startup is complete.

[0182] In some implementations, after starting the camera, the hardware layer may return information indicating that the camera startup is complete to application B so that application B can perform subsequent steps.

[0183] S206. Application B starts the preview function.

[0184] In some implementations, after receiving the information indicating that camera startup is complete, application B may enable a preview function. The preview function presents the user with the real-time image captured after the camera application is opened. After application B enables the preview function, the electronic device may trigger the capture of raw images. The electronic device triggering the capture of raw images may include steps S207-S209.

[0185] S207. Application B calls the startPreview interface to send a request to the HAL layer to capture the original image.

[0186] S208. HAL layer configuration parameters and initialization processing.

[0187] In some implementations, the HAL layer may execute S208 after receiving the request for capturing the original image sent by application B. Specifically, S208 may include the HAL layer powering on the camera, configuring parameters, and allocating streams.

[0188] Powering on the camera refers to providing power to the camera so that the camera is in a working state. For example, the HAL layer can call a corresponding power management interface to control the power switch of the camera.

[0189] Configuration parameters refer to setting various camera parameters for the camera. For example, the HAL layer passes the camera parameters set by the upper-layer application (such as application B) to the camera by calling the interface provided by the camera. In this embodiment of the application, the HAL layer can determine that the current scene is a code scanning scene based on the code scanning function activated by application B, and then configure various camera parameters for the code scanning scene.

[0190] Stream allocation refers to configuring the data transmission path and format to ensure that data can be correctly transmitted from the upper application layer and framework layer (such as the camera framework) to the camera and from the camera to the upper application for processing. For example, the HAL layer configures the data stream properties, such as data format, buffer size, transmission rate, etc., by calling the interface provided by the camera to ensure efficient transmission of the data stream.

[0191] Exemplarily, stream configuration may include configuring a preview stream and / or a data stream. Both the preview stream and the callback stream are data streams. Both the preview stream and the callback stream may include data information of the image captured by the camera. The preview stream is used for real-time display by the application. The callback stream is used for data processing and analysis by the application. Since the preview stream and the callback stream have different functions, the preview stream and the callback stream are processed differently after being returned to the application, and therefore the specific data formats of the data information in the preview stream and the callback stream may also be different. Exemplarily, the data format of the preview stream may be format33HAL_PIXEL_FORMAT_BLOB, or format34HAL_PIXEL_FORMAT_IMPLEMENTATION_DEFINED. The data format of the callback stream may be format 35HAL_PIXEL_FORMAT_YCbCr_420_888.

[0192] In the embodiment of the present application, the target image obtained by the application is used for barcode recognition. In other words, the present application mainly focuses on the callback flow scenario, that is, the original image and target image mentioned in the embodiment of the present application belong to the callback flow.

[0193] S209. The HAL layer sends a request to the hardware layer to capture the original image.

[0194] In some implementations, after executing S208 , the HAL layer may request the hardware layer to continuously acquire the original image.

[0195] In some implementations, after receiving a request to capture a raw image, the hardware layer may periodically and continuously capture the raw image. The HAL layer may also periodically generate a target image and determine whether camera parameters need to be adjusted. Specifically, the HAL layer may determine whether camera parameters need to be adjusted based on whether the current scene is a barcode scanning scene.

[0196] Exemplarily, after the hardware layer receives a request to capture the original image, step group SG21 ( S210 a - S213 a ) may be executed.

[0197] S210a. The hardware layer collects the original image m1.

[0198] S211a. The hardware layer sends the original image m1 to the HAL layer.

[0199] S212a. The HAL layer generates the target image M1 and determines whether the camera parameters need to be adjusted.

[0200] In some implementations, after receiving the original image m1, the HAL layer may generate a target image M1 and determine whether camera parameters need to be adjusted.

[0201] Specifically, after acquiring the original image m1, the HAL layer may generate the target image M1 based on the original image m1. Simultaneously, after acquiring the original image m1, the HAL layer may determine whether camera parameters need to be adjusted based on the original image m1. In other words, the HAL layer generates the target image M1 and determines whether camera parameters need to be adjusted in parallel.

[0202] In a possible implementation, the above parallel execution may include the following situations:

[0203] Case 1: The HAL layer generates the target image M1 and determines whether the camera parameters need to be adjusted. Figure 6A As shown, the HAL layer simultaneously begins generating the target image M1 and determining whether camera parameters need to be adjusted. There is no limit on when the generation of the target image M1 and the determination of whether the parameters need to be adjusted are completed. This depends on the execution efficiency of the HAL layer. For example, they can be completed simultaneously or one after the other.

[0204] Case 2: After the HAL layer starts to generate the target image M1, it starts to determine whether the camera parameters need to be adjusted before the target image M1 is completed. Figure 6BAs shown, the time when target image M1 is generated begins before the time when the camera parameter adjustment determination is started, and there is a time interval of T1. T1 is less than the time required to generate target image M1. In other words, the generation of target image M1 and the determination of whether camera parameter adjustment is necessary overlap in execution time.

[0205] Case 3: The HAL layer starts generating the target image M1 after starting to determine whether the camera parameters need to be adjusted, but before completing the determination of whether the camera parameters need to be adjusted. Figure 6C As shown, the start time is before the start time of generating the target image M1, and there is a time interval of T1. T1 is less than the time required to determine whether the camera parameters need to be adjusted. In other words, the generation of the target image M1 and the determination of whether the camera parameters need to be adjusted overlap in execution time.

[0206] For example, Figure 5 The HAL layer shown in generates the target image M1 and determines whether the camera parameters need to be adjusted. In the implementation of this application, Figure 5 The case 1 shown is only for illustrative purposes and is not intended to limit

[0207] In a possible implementation, the HAL layer may determine whether camera parameters need to be adjusted during the process of generating the target image M1.

[0208] In this embodiment of the present application, the HAL layer can determine whether camera parameters need to be adjusted based on the pre-screening module, the barcode information detection module, and the camera behavior judgment module. If it is determined that camera parameters need to be adjusted, the target camera parameters can be further obtained and the target camera parameters and corresponding adjustment instructions can be issued.

[0209] In the above S212a, the HAL determines whether the camera parameters need to be adjusted based on the original image m1, which may specifically include the following steps:

[0210] First, the HAL layer inputs the original image m1 into the pre-screening module, which predicts the area where the barcode may exist in the original image m1 and crops the area from the original image m1 to obtain the cropped image m1-1.

[0211] The HAL layer then inputs the original image m1 and the cropped image m1-1 into the barcode information detection module, which determines the characteristic information of the barcode in the original image m1. The characteristic information of the barcode in the original image m1 may include one or more of the following: the position of the barcode in the original image m1, the category of the barcode in the original image m1, the characteristic points of the barcode in the original image m1, etc. Optionally, before inputting the original image m1 and the cropped image m1-1 into the barcode information detection module, the HAL layer may also perform pre-processing on the original image m1 and the cropped image m1-1, such as grayscale processing, normalization processing, etc.

[0212] Finally, the HAL layer can input the acquired barcode feature information from the original image m1 into the camera behavior determination module. This module then determines whether camera parameters need to be adjusted based on the barcode feature information in the original image m1. If adjustment is determined to be necessary, the camera behavior determination module can then issue corresponding adjustment instructions to the hardware layer based on the determined target camera parameters.

[0213] Specifically, the zoom judgment module determines whether zooming is required based on the position and proportion of the barcode in the original image m1, and further determines the target zoom ratio when it is determined that zooming is required; the 3A judgment module determines whether 3A parameters, such as exposure parameters, focus, etc., need to be adjusted based on the feature information of the barcode in the original image m1, and further determines the target 3A parameters, such as target exposure parameters, target focus, etc. when it is determined that the 3A parameters need to be adjusted. When it is determined that the camera parameters need to be adjusted, the camera behavior judgment module can also send corresponding adjustment instructions to the hardware layer based on the determined target zoom ratio and / or target 3A parameters. Optionally, in addition to the above-mentioned focus and 3A parameters, the camera behavior judgment module can also determine whether other camera parameters need to be adjusted, which is not limited in the embodiments of the present application.

[0214] For the above-mentioned pre-screening module, barcode information detection module, and camera behavior judgment module, please refer to the subsequent more detailed introduction, which will not be repeated here.

[0215] S213a. The HAL layer sends the target image M1 to application B.

[0216] After the HAL layer generates the target image M1, S213a may be executed. Specifically, the HAL layer may send the target image M1 to the application B through the camera framework.

[0217] Optionally, if the HAL layer has obtained the feature information of the barcode in the original image m1 when executing S213a, the HAL layer can send the feature information of the target image M1 and the barcode to application B to simplify the processing flow of application B identifying the barcode in the target image M1.

[0218] It can be understood that the electronic device generates the target image M1 and determines that the camera parameters need to be adjusted through the above step group SG21.

[0219] After executing the above S210a-S213a, step group SG22 (S210b-S213b) can be executed.

[0220] S210b. The hardware layer collects the original image m2.

[0221] S211b. The hardware layer sends the original image m2 to the HAL layer.

[0222] S212b. The HAL layer generates the target image M2 and determines whether the camera parameters need to be adjusted.

[0223] S213b. The HAL layer sends the target image M2 to application B.

[0224] Similarly, after executing the above S210b-S213b, step group SG23 (S210c-S213c) is executed again. The specific steps included in step group SG22 and step group SG23 can refer to the above step group SG21 and will not be repeated here.

[0225] It is worth mentioning that Figure 5 Each step group shown is used to generate a single target image. Each of these steps is also executed once to determine whether camera parameters need to be adjusted. In other words, the electronic device generates a single target image after executing each step group. However, whether the application receives the target image sent by the HAL layer depends on whether the application is currently idle. If the application is currently idle, it receives the target image; if it is currently busy, it discards the target image. For example, application B receives target images M1 and M3, but discards target image M2.

[0226] The above step groups SG21 to SG25 are only for illustrative purposes and are not limited to Figure 5 As shown in the three frames of images, after the electronic device starts to capture the original image, it can generate multiple frames of target image. In addition, the above step group SG21-step group SG25 are executed by the hardware layer and HAL layer, and do not affect the execution operation of application B in the corresponding time period.

[0227] In some implementations, the HAL layer can determine whether camera parameters need to be adjusted. If so, the HAL layer sends an adjustment instruction to the hardware layer; if not, the HAL layer does not send an adjustment instruction to the hardware layer.

[0228] The following S214-S215 may be a case where the HAL layer determines that camera parameters need to be adjusted.

[0229] S214. The HAL layer sends an adjustment instruction to the hardware layer.

[0230] In some implementations, after the HAL layer determines whether camera parameters need to be adjusted based on the original image m1, the HAL layer determines that camera parameters need to be adjusted, further determines target camera parameters, and then executes S214. Exemplarily, the target camera parameters include target zoom magnification and / or target 3A parameters, and the adjustment instructions include zoom instructions and / or 3A adjustment instructions.

[0231] In one possible implementation, Figure 5 As shown, S214 and S213a can be executed simultaneously. In the embodiment of the present application, S214 and S213a can also be executed at different times, for example, S214 is executed after S213a, which depends on the specific situation of the above-mentioned execution of S212a.

[0232] S215. Adjust camera parameters at the hardware layer.

[0233] After receiving the adjustment instructions sent by the HAL layer, the hardware layer can adjust the camera parameters based on the adjustment instructions. For example, when the adjustment instructions include a zoom instruction, the hardware layer can adjust the camera's zoom ratio to the target zoom ratio. For example, the hardware layer can also replace the lens used based on the target zoom ratio, such as replacing it with a telephoto lens. When the adjustment instructions include a 3A adjustment instruction, the hardware layer can set the camera's 3A parameters to the target 3A parameters.

[0234] For example, Figure 5 As shown, the hardware layer completes S215 before executing S210c. That is, the hardware layer captures the original image m3 after adjusting the camera parameters. The remaining original images m1 and m2 are actually captured before adjusting the camera parameters.

[0235] In some implementations, after receiving the target image sent by the HAL layer, application B may process the target image and identify the barcode in the target image to obtain the barcode information.

[0236] S216a. Apply B to recognize the barcode in the target image M1.

[0237] When the HAL layer sends the target image M1 to the application B, since the application B is in an idle state and can process the target image M1, the application B receives the target image M1.

[0238] After receiving the target image M1, application B can recognize the barcode in the target image M1. For details, please refer to the above description of application B recognizing the barcode in the target image, which will not be repeated here.

[0239] In some implementations, after S216a is executed, the application B fails to recognize the barcode. The failure of the application B to recognize the barcode in the target image M1 may specifically include: failure to detect the position of the barcode in the target image M1, and failure to parse the barcode in the target image M1. For example, please refer to the above Figure 3A and Figure 3B The introduction of barcode recognition failure will not be elaborated here.

[0240] Optionally, after executing S216a, after parsing the barcode in the target image M1 fails, application B may issue a zoom instruction.

[0241] When application B is recognizing the barcode of target image M1, the HAL layer sends target image M2 to application B. However, since application B is busy at this time and cannot process target image M2, application B discards target image M2.

[0242] In one possible implementation, generating the target image M1 based on the HAL layer and determining whether camera parameters need to be adjusted are performed in parallel. S215 and S216a can also be performed in parallel. In other words, the hardware layer does not need to wait for application B to determine whether camera parameters need to be adjusted before adjusting the camera parameters.

[0243] S216b. Apply B to recognize the barcode in the target image M3.

[0244] When the HAL layer sends the target image M3 to the application B, since the application B is in an idle state and can process the target image M3, the application B receives the target image M3.

[0245] S217. After success, application B obtains the barcode information.

[0246] After successfully recognizing the barcode in the target image M3, application B can obtain the barcode information.

[0247] Optionally, application B may perform related operations after obtaining the barcode information. For example, the barcode information includes a link to a page, and the electronic device may jump to display the page indicated by the barcode information based on the barcode information.

[0248] In the embodiments of this application, Figure 5 The interactive process shown is for illustrative purposes only. The above-mentioned parallel execution of generating the target image M1 and determining the target camera parameters is for illustrative purposes only. It is also possible that the electronic device executes the target image M1 and determines that there is no need to adjust the camera parameters in parallel, and then executes the generation of the target image M2 and determines the target camera parameters in parallel. When the electronic device executes the code scanning method provided in the embodiment of the present application, there may be more situations based on actual circumstances, and there is no limitation on this.

[0249] from Figure 5 The timing diagram shows that the HAL layer can directly determine whether the camera parameters need to be adjusted based on the original image. There is no need to wait until application B fails to recognize the barcode in the target image before determining whether the camera parameters need to be adjusted. In this way, the hardware layer only needs to wait for a short period of time to complete the camera parameter adjustment, and then the barcode can be successfully recognized.

[0250] Specifically, the HAL layer determines whether camera parameters need to be adjusted for the original image in frame t and issues an adjustment instruction. At this point (i.e., when the adjustment instruction is issued), the application layer attempts to identify the barcode in the target image in frame t and fails to do so. After receiving the adjustment instruction, the hardware layer adjusts the camera parameters. Since adjusting camera parameters also takes some time, after completing the adjustment, the hardware layer can send the adjusted original image to the HAL layer. The HAL layer then sends the adjusted target image to application B. At this point (i.e., when application B receives the adjusted target image), it is already frame t+n. Here, n can be the ratio of the hardware layer adjustment time to the single-frame time. Therefore, if application B fails to recognize the barcode in frame t, it only needs to wait until it receives frame t+n to successfully recognize the barcode. This shortens the overall cycle time and improves the user scanning experience.

[0251] The following introduces Figure 5 The method flow shown is relative to the above Figure 2 The method flow shown has more beneficial effects. Figure 7A and Figure 7B For example, a single frame duration of 30ms is used as an example for introduction. Figure 7A and Figure 7B The time axis shown represents the time when the target image was generated. For example, "t" on the time axis indicates the time when the t-th frame of the target image was generated, "t+1" on the time axis indicates the time when the t-th frame of the target image was generated, and so on.

[0252] Figure 7A This example shows how to perform the above Figure 2 The timing diagram of the code scanning method shown in FIG.

[0253] refer to Figure 7AApplication A receives the target image frame t and begins recognizing the barcode therein. 50 ms later, Application A fails to recognize the barcode therein and issues a zoom command. Within 50 ms of recognizing the barcode therein, Application A discards the target image frame t+1. The hardware layer completes the zoom process in 25 ms after receiving the zoom command. Within 25 ms of performing the zoom process at the hardware layer, Application A receives the target image frame t+2 and begins recognizing the barcode therein. 50 ms later, Application A fails to recognize the barcode therein. Within 50 ms of recognizing the barcode therein, Application A discards the target image frame t+3. After completing the zoom process, the hardware layer generates the original image frame t+4. Application A receives the target image frame t+4 and begins recognizing the barcode therein. 50 ms later, Application A successfully recognizes the barcode therein.

[0254] That is to say, in Figure 7A In the timing diagram shown in FIG, after the barcode in the target image of the t-th frame fails to be recognized, the barcode in the target image of the t+4th frame can be recognized successfully at the earliest. For example, Figure 7A In the example, after application A fails to recognize the barcode twice, application A can only successfully recognize the barcode for the third time. Figure 7A In the example, it takes at least 160ms to successfully recognize the barcode from the start of recognizing the barcode in the target image of the tth frame.

[0255] Figure 7B This example shows how to perform the above Figure 5 The timing diagram of the code scanning method shown in FIG.

[0256] refer to Figure 7BThe HAL layer concurrently generates the target image frame t and determines whether camera parameters need to be adjusted. For example, the HAL layer's pre-screening module, barcode information detection module, and camera behavior judgment module determine whether camera parameter adjustment is necessary. Application B receives the target image frame t from the HAL layer and begins recognizing the barcode in it. 50 ms later, Application B fails to recognize the barcode in the target image frame t. Within 50 ms of recognizing the barcode in the target image frame t, Application B discards the target image frame t+1. The hardware layer receives the adjustment instructions from the HAL layer and begins adjusting the camera parameters. 25 ms later, the hardware layer completes the camera parameter adjustment. The moment Application B begins recognizing the items in the target image frame t coincides with the moment the hardware layer begins adjusting the camera parameters. After completing the camera parameter adjustment, the hardware layer generates the original image frame t+2. Application B receives the target image frame t+2 and begins recognizing the barcode in it. 50 ms later, Application B successfully recognizes the barcode in the target image frame t+2.

[0257] That is to say, in Figure 7B In the timing diagram shown, after the barcode in the target image of frame t fails to be recognized, the barcode in the target image of frame t+2 can be recognized successfully at the earliest. Figure 7B In the example, after application B fails to recognize the barcode once, application B may succeed in recognizing the barcode the second time. Figure 7B In the process, from the start of recognizing the barcode of the target image in frame t, it takes at least 100ms to successfully recognize the barcode.

[0258] It is understandable that, when the barcode in the target image obtained after adjusting the parameters is successfully recognized, Figure 7B The scanning process shown can be compared Figure 7A The barcode scanning process shown here successfully recognizes the barcode within X frames, where X can be the smallest integer greater than or equal to the ratio of the application's barcode recognition duration to the single-frame duration. For example, if the application's barcode recognition duration is 50ms and the single-frame duration is 30ms, and the smallest integer greater than the ratio of 50 to 30 is 2, then X is 2.

[0259] above Figure 7A and Figure 7B The specific processing times for barcode recognition and zoom processing mentioned in the text, such as 50ms and 25ms, are only for illustrative purposes. In the embodiments of the present application, considering various factors such as the performance of the electronic device, the specific processing times for barcode recognition and zoom processing may also be other times, and there is no limitation on this.

[0260] Combined with the above Figures 2 to 7B From the embodiment shown, it can be seen that: Figure 2The barcode scanning method shown requires that after Application A fails to recognize the barcode on a frame of the target image, it then issues a zoom command to the hardware layer to perform the zoom process. This not only causes Application A to discard multiple frames of the target image during the recognition process, wasting resources, but also requires the hardware layer to wait a long time before adjusting the camera parameters. Application A can only successfully recognize the barcode on the target image after the camera parameters are adjusted. This excessively long barcode recognition time results in a low recognition success rate.

[0261] Figure 5 The barcode scanning method shown does not need to wait until application B fails to recognize the barcode in the target image before determining whether the camera parameters need to be adjusted. The HAL layer can directly determine whether the camera parameters need to be adjusted based on the original image. In this way, the hardware layer only needs to wait for a short period of time to complete the adjustment of the camera parameters. Application B successfully recognizes the barcode of the target image after the camera parameters are adjusted. The time for barcode recognition is short, and the recognition success rate is high.

[0262] The following combination Figure 5 The barcode position detection module, pre-screening module, and camera behavior judgment module provided in the embodiments of the present application are introduced.

[0263] In some implementations, the barcode information detection module is used to determine characteristic information of a barcode in an original image. The characteristic information of a barcode includes one or more of the following: the position, category, and characteristic points of the barcode. The barcode information detection module includes modules such as a barcode position detection module and a barcode characteristic point detection module.

[0264] The barcode position detection module is used to detect the original image and obtain the position and category of the barcode in the original image.

[0265] In the embodiments of the present application, the position of the barcode can be represented by a variety of methods or algorithms, which are not limited to this. For example, the position of the barcode can be represented by a bounding box. For example, the position of the barcode can be represented by the coordinates of the upper left corner of the barcode (x1, y1) and the coordinates of the lower right corner (x2, y2). Bounding box is an algorithm for finding the optimal bounding space of a discrete point set. The basic idea is to use a slightly larger geometric body with simpler characteristics to approximately replace a complex geometric object.

[0266] In an embodiment of the present application, the barcode position detection module can use a variety of specific detection methods to obtain the position of the barcode from the original image, which is not limited. The methods for detecting the position of the barcode include traditional methods and methods based on neural network models. Exemplarily, the above-mentioned traditional method refers to determining the position of the barcode by identifying the edge of the barcode. The above-mentioned neural network-based method refers to automatically learning the complex features in the original image through a neural network model to determine the position of the barcode. Exemplarily, the barcode position detection module can determine the position of the barcode in the original image through a neural network model whose network structure is a cross-stage partial path aggregation network (Cross Stage Partial Path Aggregation Network, CSP-PAN).

[0267] Figure 8 The schematic diagram of the neural network model based on the CSP-PAN structure provided in the embodiment of the present application is exemplified.

[0268] refer to Figure 8 The neural network model used to determine the barcode location can include a backbone network (Backbone), a CSP-PAN structure, and a head. The backbone is the network in the neural network model used to extract key features. The backbone is typically composed of a pre-trained convolutional neural network, such as a Residual Network (ResNet) or a Visual Geometry Group Network (VGG). The CSP-PAN structure is a feature fusion module in the field of object detection, which is an improvement on the Path Aggregation Network (PANet). The CSP module is a feature processing concept that divides the feature map into two parts: one part undergoes convolution operations, and the other part is concatenated with the convolution results. This reduces computational complexity and improves feature utilization. The PAN structure adds a top-down feature structure to the FPN to achieve more comprehensive feature fusion. The head, as the output of the neural network model, is responsible for performing specific task predictions based on the fused feature information, such as object detection and image segmentation. For example, in the object detection task, the head includes a classification head and a regression head, which are used to predict the object category and the position of the bounding box, respectively.

[0269] Exemplarily, the original image is input into the Backbone in the neural network model. The Backbone extracts features of different levels and scales from the original image, generates feature information of the original image, and inputs the feature information of the original image into the CSP-PAN structure. The CSP-PAN structure receives the feature information of the original image input by the Backbone, fuses the features through different paths to obtain the feature information of the fused original image, and inputs the feature information of the fused original image into the subsequent network Head. The Head receives the feature information of the fused original image sent by the CSP-PAN structure, determines the position of the barcode in the original image through the regression head, and optionally, the Head can also determine the category of the barcode in the original image through the classification head.

[0270] In the embodiment of the present application, the above-mentioned method of using a neural network model based on the CSP-PAN structure to obtain the position and category of the barcode is only for illustrative purposes, and neural network models of other structures can also be used, without limitation.

[0271] In some implementations, an original image may contain one barcode, may contain multiple barcodes, and may also contain other barcode-like graphics. The barcode position detection module can detect multiple areas that may contain barcodes. In this case, further, the barcode position detection module can also determine the confidence of the multiple detected areas that may contain barcodes. The barcode position detection module is provided with a confidence threshold, and when the confidence of the area that may contain the barcode exceeds the confidence threshold, the confidence will be regarded as a valid confidence. Further, among the multiple areas that may contain barcodes detected, the barcode contained in the area whose confidence exceeds the confidence threshold is determined to be a valid barcode. In addition, the barcode contained in the area with the highest confidence is determined to be the barcode in the original image.

[0272] The above-mentioned multiple areas that may contain barcodes can be understood as areas that the barcode position detection module believes may contain barcodes in the original image, but are not necessarily areas that actually contain barcodes in the original image.

[0273] Optionally, the barcode position detection module can also determine the type of the barcode. Specifically, the barcode position detection module can determine whether the barcode contained in the original image is a one-dimensional barcode or a two-dimensional barcode. Further, after determining that the barcode is a two-dimensional barcode, the module can determine the specific type of the two-dimensional barcode. For example, the barcode position detection module can calculate the confidence level that a detected barcode belongs to each barcode type and determine the barcode type with the highest confidence level as the barcode type.

[0274] The barcode feature point detection module detects the original image and obtains the coordinates of the barcode's feature points. Specifically, after the barcode position detection module processes the original image, the barcode's position within the original image is determined. The barcode information detection module crops the area containing the barcode in the original image to produce a cropped original image. This cropped original image is then input into the barcode feature point detection module to obtain the coordinates of the barcode's feature points.

[0275] In the embodiments of the present application, a characteristic point of a barcode may specifically refer to a certain point or points in the barcode. For example, taking a QR code as an example, a characteristic point may refer to a corner point of the QR code; a characteristic point may also refer to a calibration point of the QR code; or a characteristic point may also refer to a corner point of a positioning pattern in the QR code.

[0276] Figure 9A The schematic diagram of the morphological structure of the QR Code provided in the embodiment of the present application is exemplified.

[0277] refer to Figure 9A , Figure 9A (a) shows the morphological structure of the QR Code, which includes three positioning patterns at the upper left corner, upper right corner, and lower left corner, as well as a calibration pattern at the lower right corner.

[0278] Figure 9A (b) shows an example of a feature point of a QRCode. The feature point of a QRCode refers to the corner points of the QRCode, i.e., the lower left corner, upper left corner, upper right corner and lower right corner of the QRCode, for example, Figure 9A Points 1, 2, 3 and 4 are shown in (b).

[0279] Figure 9A (c) shows another exemplary feature point of a QR Code. The feature point of a QR Code refers to the calibration point of the QR Code, that is, the center point of the calibration graphic, for example, Figure 9A Point 1 shown in (c).

[0280] Figure 9A (d) shows another characteristic point of QRCode. The characteristic point of QRCode refers to the corner points of the positioning pattern in QRCode, that is, the lower left corner, upper left corner, upper right corner and lower right corner of the positioning pattern in QRCode, for example, Figure 9A Point 1, point 2, point 3, point 4, point 5, point 6, point 7, point 8, point 9, point 10, point 11 and point 12 shown in (d).

[0281] In the embodiments of this application, Figure 9AThe characteristic points of the barcode shown are for illustrative purposes only and may also refer to other points in the barcode, without limitation.

[0282] In an embodiment of the present application, the barcode position detection module can use a variety of specific detection methods to obtain the feature points of the barcode from the original image, which is not limited. The methods for detecting barcode feature points include traditional methods and methods based on neural network models. The above-mentioned neural network-based method refers to automatically learning the complex features in the original image through a neural network model to determine the feature points of the barcode. Exemplarily, the barcode feature point detection module can obtain a heat map of the four corner points of the QR Code and determine the four points with the strongest response and greater than a threshold as the feature points of the barcode.

[0283] In a possible implementation, the feature points of the barcode acquired by the barcode position detection module may be as follows: Figure 9A (b) Figure 9A (c) and Figure 9A The feature points shown in (d) can also be Figure 9B The feature points shown in (a) are as follows. This is because: the angle between the plane of the physical QR code and the camera of the electronic device is small, that is, when the user scans the code, the electronic device is not facing the physical barcode, for example, the upper or lower end of the electronic device is closer to the physical barcode, or the left or right end of the electronic device is closer to the physical barcode. In this scanning scenario, the original image collected by the electronic device will be distorted, for example, the barcode will appear trapezoidal or barrel-shaped during imaging, and then the feature points of the barcode obtained by the barcode position detection module based on the original image will be as follows Figure 9B As shown in (a). Figure 9B The feature points in (a) are the focus of the barcode, i.e., the lower left corner, upper left corner, upper right corner and lower right corner of the barcode, for example, Figure 9B Points 1, 2, 3 and 4 are shown in (a). It can be seen that: Figure 9B The barcode composed of the feature points in (a) is distorted.

[0284] Electronic devices obtain Figure 9B After the feature points of the barcode shown in (a) are obtained, the feature points of the barcode can be sent to the application side, which performs correction processing to restore the feature points of the barcode in the original image captured when no distortion occurs, for example Figure 9B Specifically, the correction process can generally be performed by a distortion correction algorithm, image enhancement technology, etc. In the embodiment of the present application, the above correction process is not limited.

[0285] Figure 10 The following is a schematic diagram illustrating a process for obtaining characteristic points of a barcode provided in an embodiment of the present application.

[0286] refer to Figure 10 , convert the cropped original image into a digital form, such as a pixel matrix input into the encoder (Encode) Encode. Encode can process the cropped original image and extract the features of the cropped original image, such as point feature maps containing the features in the cropped original image. Encode can usually be a convolutional neural network. The points feature maps are then input into the linear layer (Linear Layer). As a neural network layer, the Linear Layer can convert the points feature maps into a one-dimensional heat map (1D Heatmap) through matrix multiplication. Each value in the heat map represents the feature intensity or response intensity of the corresponding position. The peak position in each heat map may correspond to the feature point of the barcode in the cropped original image. Finally, by analyzing the 1D Heatmap, the four points with the strongest response and greater than the threshold can be determined, which are the feature points of the barcode.

[0287] In the embodiment of the present application, the above Figure 10 The method of obtaining the characteristic points of the barcode shown is only for illustrative purposes, and other methods may also be used to obtain the characteristic points of the barcode, which is not limited thereto.

[0288] In the embodiment of the present application, the barcode information detection module is deployed in the HAL layer. The HAL layer can obtain barcode feature information such as the barcode's location, category, and feature points. The HAL layer can determine whether to adjust camera parameters based on this feature information. It can also send this feature information to the corresponding application at the same time as the original image is sent to the corresponding application, thereby reducing application processing time and improving application processing efficiency.

[0289] The pre-screening module is used to predict the area where the barcode may exist in the original image. Specifically, the pre-screening module can crop the area where the barcode may exist in the original image before the barcode information detection module is executed to obtain a cropped original image. This is because if the original image is directly input into the barcode information detection module, when the proportion of the barcode in the original image is less than a certain proportion (such as 20%), the barcode information detection module may not be able to determine the position of the barcode, which will give the barcode information detection module the illusion that the original image does not contain a barcode, thereby affecting the subsequent adjustment of the camera parameters. Therefore, the pre-screening module can also be understood as being used to increase the proportion of the barcode in the image and to increase the success rate of the barcode information detection module in obtaining the characteristic information of the barcode.

[0290] In the embodiments of the present application, the pre-screening module can use a variety of specific methods to crop the area in the original image where the barcode may be present to obtain a cropped original image, which is not limited to this method. Methods for obtaining the cropped image include manually designed anchor box methods and machine learning-based methods, such as a machine learning-based salient region detection algorithm.

[0291] The following describes a method flow for obtaining a cropping area based on a priori frame in a pre-screening module provided in an embodiment of the present application.

[0292] Step a: Set the prior box.

[0293] Prior boxes (or anchor boxes) are used to improve the accuracy of target location and classification. Prior boxes are a set of rectangular boxes of varying sizes and proportions preset on an image, used to match and predict possible targets within the image. There can be one or more prior boxes. The size, dimensions, and position of the prior boxes can be generated using a clustering algorithm (such as the K-means algorithm) or manually set; there are no restrictions on this.

[0294] Figure 11A A schematic diagram of setting a priori frame provided in an embodiment of the present application is exemplarily shown.

[0295] refer to Figure 11A , the pre-screening module sets a total of 5 prior frames for the original image, namely prior frame 1, prior frame 2, prior frame 3, prior frame 4, and prior frame 5. For example, Figure 11A The gray rectangle in the image can be regarded as a barcode in the original image.

[0296] In the embodiment of the present application, the above Figure 11A The pre-screening frame is for illustrative purposes only. The pre-screening module can also set pre-screening frames of other data and sizes without limitation. For example, based on the user's tendency to center the barcode when scanning, the pre-screening module may set a pre-screening frame centered in the original image and 1 / 6 the size of the original image.

[0297] Step b-1: for the first frame of the original image, the pre-screening module determines the priori frame located at the center of the original image from the set multiple priori frames as the final selected priori frame.

[0298] This is because the original image is obtained for the first time and there is no other original image for reference. Therefore, the center area of ​​the default original image may be the area where the barcode is located.

[0299] Step b-2: for the t-th frame original image, the pre-screening module determines the final selected priori frame from the set multiple priori frames based on the barcode position detection result of the t-1-th frame original image output by the barcode information detection module.

[0300] It is understood that during execution, the barcode information detection module can determine that there are multiple areas in the original image that may contain barcodes. The barcode information detection module can also determine the confidence levels of the multiple areas that may contain barcodes. Among them, the barcodes contained in the areas with confidence levels higher than the confidence threshold are determined by the barcode information detection module to be valid barcodes. In other words, the barcode position detection result includes the confidence levels of multiple areas in the original image of the t-1th frame that may contain barcodes (including: the confidence levels of areas containing valid barcodes and the confidence levels of areas containing invalid barcodes).

[0301] In step b-2, the pre-screening module obtains multiple regions in the image frame t-1 that may contain a barcode and the confidence levels of these regions, and assigns these regions to various prior frames. For example, the allocation can be determined by calculating a cost matrix between the regions and the prior frames. This translates into an optimal transmission problem.

[0302] Figure 11B The schematic diagram of the area where the possible barcode is located in the original image of the t-1 frame is shown as an example.

[0303] refer to Figure 11B , the pre-screening module sets two prior frames for the original image of the t-1th frame, namely prior frame 1 and prior frame 2. Figure 11B The gray rectangle in the figure is the area in the original image of the t-1th frame that may contain a barcode detected by the bar detection information module. For example, the area that may contain a barcode includes box1, box2, box3, box4, box5 and box6. Among them, box1, box2 and box3 fall into the priori box, and box4, box5 and box6 do not fall into the priori box. The pre-screening module can assign box1, box2 and box3 located in the priori box to priori box 1 and priori box 2 respectively, that is, it can be predicted whether the above-mentioned box1, box2 and box3 are most likely to be located in priori box 1 or priori box 2. Exemplarily, the cost matrix of box1, box2 and box3 and priori box 1 and priori box 2 can be calculated by IOU Loss and L2Loss.

[0304] Among them, Intersection over Union Loss (IOU Loss) is a loss function commonly used in object detection tasks. It is used to measure the degree of overlap between the predicted bounding box (such as the area that may contain a barcode) and the true bounding box (such as the prior box). Mean Squared Error Loss (L2 Loss) measures the model's prediction error by calculating the average of the squares of the differences between the predicted value and the true value. For example, L2 Loss is used to measure the distance between the area that may contain a barcode and the center point of the prior box.

[0305] cost=w dis L dis +w iou L iou (1)

[0306] Among them, formula (1) is a weighted loss function used to calculate the cost of the area that may contain the barcode and the prior box. The pre-screening module assigns the area that may contain the barcode to the prior box with lower cost. dis and L iou Represent different loss terms. dis Indicates L dis The weight, w iou Indicates L iou The weight is used to adjust L dis and L iou The specific gravity. L dis It is used to measure the distance between the area that may contain the barcode and the center point of the prior frame, which can be calculated by L2 Loss. iou It is used to measure the degree of overlap between the area that may contain the barcode and the prior frame, which can be calculated by IOU Loss.

[0307]

[0308] Among them, formula (2) is used to calculate L dis . Represents the relevant data of the area that may contain barcodes obtained by the pre-screening module, such as the coordinates of the area, the width and height of the area, etc. N represents the number of categories of relevant data of the area that may contain barcodes. i Represents the relevant data of the prior frame, such as the coordinates of the prior frame, the width and height of the prior frame, etc. dis The smaller the value, the smaller the distance between the area that may contain the barcode and the center point of the prior box.

[0309]

[0310] L iou=1-IOU(4)

[0311] Among them, formula (3) and formula (4) are used to calculate L iou . AreaofOverlap represents the intersection area of ​​the area that may contain the barcode and the prior frame, that is, the area of ​​the overlapping part of the area that may contain the barcode and the prior frame. AreaofUnion represents the union area of ​​the area that may contain the barcode and the prior frame, that is, the total area covered by the area that may contain the barcode and the prior frame, including the overlapping part. The value range of IOU is between 0 and 1. The larger the IOU value, the higher the degree of overlap between the area that may contain the barcode and the prior frame. When IOU is 1, it means that the area that may contain the barcode completely overlaps with the prior frame; when IOU is 0, it means that the area that may contain the barcode does not overlap with the prior frame. L iou The value range is between 0 and 1. iou The smaller the value of , the higher the overlap between the area that may contain the barcode and the prior box.

[0312] cost box1 box2 box3 Prior box 1 0.7 0.8 1.4 Prior frame 2 0.1 0.4 0.3

[0313] Table 1

[0314] Table 1 shows an example Figure 11B The costs of box1, box2 and box3 with the priori box 1 and priori box 2. For example, the costs of box1, box2 and box3 with the priori box 1 and priori box 2 are calculated respectively by the above formula (1). Among them, the cost between box1 and the priori box 1 is 0.7, and the cost between box1 and the priori box 2 is 0.1. The pre-screening module can assign box1 to the priori box 2 with lower cost. The cost between box2 and the priori box 1 is 0.8, and the cost between box2 and the priori box 2 is 0.4. The pre-screening module can assign box2 to the priori box 2 with lower cost. The cost between box3 and the priori box 1 is 1.4, and the cost between box3 and the priori box 2 is 0.3. The pre-screening module can assign box3 to the priori box 2 with lower cost.

[0315] In the embodiment of the present application, the algorithm or method used to calculate the cost matrix of multiple possible barcodes and prior frames is only for illustrative purposes, and other methods and algorithms can also be used for calculation, without limitation.

[0316] After assigning the areas that may contain barcodes to each prior frame, the score of each prior frame can be calculated. The score of the prior frame is the sum of the confidences of the areas that may contain barcodes assigned to the prior frame:

[0317]

[0318] Among them, formula (5) is used to calculate the score of the prior frame. n Represents the confidence of an area that may contain a barcode assigned to the priori box. K represents the number of areas that may contain barcodes assigned to the priori box. For example, from Table 1 above, it can be seen that no area that may contain a barcode is assigned to priori box 1, and box1, box2, and box3 are assigned to priori box 2. According to formula (5), the score of priori box 1 is 0, and the score of priori box 2 is the sum of the confidences of box1, box2, and box3. The score of priori box 2 is higher than that of prior box 1, that is, the priori box finally selected is priori box 2.

[0319] In the embodiment of the present application, the above-mentioned method of determining the final selected prior frame from the set multiple prior frames is only for illustrative purposes. In step b-2, the final selected prior frame can also be determined by more other methods, which are not limited to this.

[0320] Step c: In the original image, the area where the final selected prior frame is located is cropped to obtain a cropped original image.

[0321] The pre-screening module may perform step c after performing the above step b-1 or the above step b-2.

[0322] Figure 11C The diagram exemplarily shows the input information and output information of the barcode information detection module provided in the embodiment of the present application.

[0323] refer to Figure 11C The original image and the cropped original image can be input into the barcode information detection module. The barcode information detection module processes the original and cropped original images to obtain barcode feature information, such as the barcode location, feature points, and category. Optionally, the original and cropped images can be pre-processed by grayscaling, normalizing, or reducing resolution before input into the barcode information detection module. Grayscaling refers to the process of converting a color image into a grayscale image. This reduces color information in the original image, simplifies the data image, and improves computational efficiency. Specifically, grayscaling can be performed using the average or weighted average method. Normalization adjusts image data to a specific range, typically adjusting the pixel values ​​from [-255, 255] to [-1, 1]. Reducing the resolution can reduce file size and transmission bandwidth. For example, reducing the image resolution from 1080p to 300 pixels * 300 pixels can reduce the computational effort of the barcode information detection module and expedite the acquisition of barcode feature information from the original image.

[0324] In the embodiment of the present application, the pre-screening module is deployed in the HAL layer. The HAL layer can crop the area in the current frame of the original image where the barcode may be located based on the barcode position detection result in the previous frame of the original image, thereby obtaining a cropped original image. This increases the proportion of the barcode in the image, improves the success rate of the barcode information detection module in obtaining the barcode feature information, and increases the maximum recognition distance of the barcode.

[0325] In some implementations, the camera behavior determination module is used to determine whether camera parameters need to be adjusted and whether corresponding adjustment instructions need to be sent to the hardware layer. The camera behavior determination module includes: a zoom determination module, a 3A determination module, and other modules.

[0326] Among them, the zoom judgment module is used to determine whether zooming is required and to determine the target zoom ratio. Specifically, the zoom judgment module can determine whether zooming is required based on the position and proportion of the barcode in the original image m1. Exemplarily, if the proportion of the barcode in the original image is less than 20%, and / or the barcode is incomplete in the original image, zooming is required; the target zoom ratio is a zoom ratio that can expand the proportion of the barcode in the original image to 20%. In the embodiment of the present application, it is not limited to the above-mentioned proportion of 20%, and other proportions can also be used to determine whether zooming is required, and there is no limitation on this.

[0327] The 3A judgment module is used to judge whether the 3A parameters need to be adjusted and to determine the target 3A parameters.

[0328] The 3A judgment module can determine the target focus parameter in the following way: when the barcode information detection module detects the presence of a barcode in the original image for the first N consecutive frames, the center point coordinates of the barcode in the original image of the current frame can be set as the focus of the camera. Specifically, the counter focuscounter is set to a value of 0. If the IOU value of the barcode in the original image of the current frame and the barcode in the original image of the previous frame is greater than a preset value, such as 0.5, the counter focuscounter is increased by 1. At this time, the IOU value of the barcode in the original image of the current frame and the barcode in the original image of the previous frame refers to the ratio of the intersection area of ​​the barcode in the original image of the current frame and the barcode in the original image of the previous frame, and the union area of ​​the barcode in the original image of the current frame and the barcode in the original image of the previous frame. When the counter focuscountor is greater than the preset number of times, for example 5 times, the 3A judgment module calls the setFocusPoint interface to set the center point coordinates (x, y) of the barcode in the original image of the current frame as the focus of the camera, that is, the target focus parameter.

[0329] In the embodiment of the present application, the above-mentioned method of determining the target focus parameter is only for illustrative purposes, and the target focus parameter can also be determined by other methods, which are not limited to this.

[0330] The 3A judgment module can determine the target exposure parameter by the following method: determining the target exposure parameter by judging the gray value of the area where the barcode is located in the original image of the current frame. Figure 11D ,exist Figure 11D In the original image, the width is a and the height is b. The width of the area where the barcode is located in the original image is x and the height is y, and the barcode is located in the area from 1 / 3a to 2 / 3a and from 1 / 3b to 2 / 3b in the original image. In this case, calculate Figure 11D The grayscale values ​​within the medium-gray rectangular area D are calculated as follows: the grayscale values ​​within the areas 1 / 4x to 3 / 4x and 1 / 4y to 3 / 4y within the barcode area. Alternatively, if the barcode is not located within the areas 1 / 3a to 2 / 3a and 1 / 3b to 2 / 3b in the original image, the grayscale values ​​within the areas 1 / 3a to 2 / 3a and 1 / 3b to 2 / 3b in the original image are calculated. When the grayscale value in area D is less than or equal to 50, the target exposure parameter is set to 2; when the grayscale value in area D is greater than 50 and less than or equal to 90, the target exposure parameter is set to 1; when the grayscale value in area D is greater than 90 and less than or equal to 140, the target exposure parameter is set to 0; when the grayscale value in area D is greater than 140 and less than or equal to 190, the target exposure parameter is set to -1; when the grayscale value in area D is greater than 190, the target exposure parameter is set to -1. The 3A judgment module calls the corresponding camera interface to adjust the exposure parameters of the hardware layer to the target exposure parameters.

[0331] In the embodiment of the present application, the above-mentioned method of determining the target exposure parameters is only for illustrative purposes. The target exposure parameters may also be determined by other methods, which are not limited thereto.

[0332] The following describes a code scanning method provided in an embodiment of the present application.

[0333] Figure 12 The following is an example flow chart of a barcode scanning method provided by an embodiment of the present application. The method is applied to an electronic device having a HAL layer including a barcode position detection module, a pre-screening module, and a camera behavior judgment module. The method includes:

[0334] S301: In response to starting a scanning function of a scanning application, the electronic device uses a first camera parameter through a hardware layer to capture a first original image, where the first original image includes a barcode.

[0335] For example, the code scanning application may refer to the above Figure 5 In application B, when the first original image is the first frame image captured after the electronic device starts a camera, the first camera parameter may be a default camera parameter.

[0336] S302: The electronic device generates a first target image based on the first original image through the HAL layer, and concurrently performs an action of determining first target camera parameters based on the first original image through the HAL layer.

[0337] The above parallel execution can refer to Figures 6A to 6C The schematic diagram shown.

[0338] In one possible implementation, after executing S302, the electronic device may further transmit the first target image to the barcode scanning application through the HAL layer. Then, the barcode in the first target image is recognized by the barcode scanning application; wherein, when the barcode in the first target image is recognized by the barcode scanning application, the electronic device may further adjust the camera parameters of the electronic device to the first target camera parameters through the hardware layer. For example, referring to Figure 5 The two steps S216a and S215 in .

[0339] Specifically, in response to generating a first target image based on the first original image through the HAL layer, the electronic device sends the first target image to the code scanning application through the HAL layer.

[0340] S303: The electronic device transmits the first target camera parameter to the hardware layer through the HAL layer.

[0341] Specifically, the electronic device transmits the first target camera parameters to the hardware layer through the HAL layer in response to analyzing the first original image through the HAL layer. Exemplarily, the HAL layer determines that the camera parameters need to be adjusted based on the first original image and outputs the first target camera parameters.

[0342] In one possible implementation, S303 may specifically include: the electronic device obtaining, via the HAL layer, first feature information of a barcode in the first original image. Exemplarily, the first feature information may be obtained by a barcode information detection module in the HAL layer. The electronic device then determines, via the HAL layer, first target camera parameters based on the first feature information of the barcode in the first original image. Exemplarily, the first target camera parameters may be obtained by a camera behavior determination module in the HAL layer.

[0343] In a possible implementation, before obtaining the first feature information of the barcode in the first original image through the HAL layer, the electronic device may further crop the first original image through the HAL layer to obtain a first cropped image, where the first cropped image includes the barcode. The electronic device may obtain the first feature information of the barcode in the first original image through the HAL layer based on the first original image and the first cropped image, for example, referring to the above Figure 11C For example, you can Figure 5 The pre-screening module shown in crops the original image m1 to obtain a cropped original image.

[0344] Specifically, the electronic device determines the first a priori frame from the preset a priori frame based on the characteristic information of the barcode in the third original image through the HAL layer, and the third original image is collected before the first original image. Then, the electronic device can crop the area where the first a priori frame is located in the first original image through the HAL layer to obtain a first cropped image. For example, please refer to the above introduction on the pre-screening module and the introduction on Figure 11A and Figure 11B Related introduction.

[0345] In one possible implementation, the first target camera parameter includes a first target zoom factor. S303 may specifically include: the electronic device determining, via the HAL layer, a first proportion of the barcode in the second original image. The electronic device then determines, via the HAL layer, the first target zoom factor based on the first proportion. When the position of the electronic device remains unchanged, the proportion of the barcode in the original image captured using the first target zoom factor is greater than the first proportion. For example, reference may be made to the above description of the zoom determination module.

[0346] In one possible implementation, the first target camera parameter includes a first target exposure parameter. S303 specifically includes: the electronic device determining, through the HAL layer, a first grayscale value of all or part of a region where the barcode is located in the second original image. The electronic device then determines, through the HAL layer, the first target exposure parameter based on the first grayscale value.

[0347] In a possible implementation, the first target camera parameter includes a first target focus parameter, and the above S303 specifically includes:

[0348] The electronic device determines the first coordinate of the center of the barcode in the second original image through the HAL layer. The electronic device then determines the first target focus parameter based on the first coordinate through the HAL layer. For example, reference can be made to the above-mentioned 3A judgment module and Figure 11D Related introduction.

[0349] S304: The electronic device uses the first target camera parameters to capture a second original image through a hardware layer, where the second original image includes a barcode.

[0350] For example, the second original image may refer to Figure 5 In the embodiment of the present application, whether the first original image and the second original image are two consecutive frames of original images is not limited, which is affected by adjusting the camera parameters at the hardware layer.

[0351] S305: The electronic device generates a second target image based on the second original image through the HAL layer.

[0352] In one possible implementation, the electronic device may also obtain second feature information of the barcode in the second original image through the HAL layer, wherein the action of obtaining the second feature information of the barcode in the second original image through the HAL layer and the action of generating a second target image based on the second original image through the HAL layer are performed in parallel.

[0353] S306: The electronic device obtains information carried by the barcode in the second target image through a barcode scanning application.

[0354] In a possible implementation, the information carried by the barcode in the second target image is acquired according to the second feature information through a barcode scanning application.

[0355] The second feature information includes information such as the position, category, and feature points of the barcode in the second target image.

[0356] In one possible implementation, the second feature information includes the first feature point of the barcode in the second original image. The first feature point is in a distorted state. The electronic device can correct the first feature point to obtain the corrected first feature point. The electronic device can also obtain the information carried by the barcode in the second target image based on the corrected first feature point. For example, you can refer to the above Figure 9B The schematic diagram shown.

[0357] In one possible implementation, the electronic device may further capture a third original image using the first camera parameters through the hardware layer. The electronic device generates a third target image based on the third original image through the HAL layer. The electronic device then discards the third target image through the code scanning application. The third original image is captured after the first original image and before the second original image. For example, the third original image may refer to Figure 5 The original image m2 shown in FIG, the third target image can refer to Figure 5 The target image M2 is shown in .

[0358] In one possible implementation, if the first target image is Figure 5 If the target image M2 is shown in , the electronic device discards the first target image through the code scanning application.

[0359] Implementation of the above Figure 12 The method flow shown does not require waiting until the application fails to recognize the barcode in the target image before determining whether the camera parameters need to be adjusted. The HAL layer can directly determine whether the camera parameters need to be adjusted based on the original image. In this way, the hardware layer only needs to wait for a short period of time to complete the adjustment of the camera parameters. The application successfully recognizes the barcode of the target image after the camera parameters are adjusted. The time for barcode recognition is short, and the recognition success rate is high.

[0360] Furthermore, the HAL layer's determination of whether camera parameters need to be adjusted does not affect the application's acquisition of the target image. In other words, the HAL layer can generate the target image and determine whether camera parameters need to be adjusted in parallel. This allows the application to display the acquired target image at a normal frame rate and recognize the QR code within the target image.

[0361] Figure 13 A schematic structural diagram of the electronic device 100 is shown.

[0362] The embodiment will be described in detail below using the electronic device 100 as an example. It should be understood that Figure 13 The electronic device 100 shown is only one example, and the electronic device 100 may have more Figure 13 The more or less components shown in the figure can be combined with two or more components, or can have different component configurations. The various components shown in the figure can be implemented in hardware, software, or a combination of hardware and software including one or more signal processing and / or application specific integrated circuits.

[0363] Exemplarily, the electronic device 100 may be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a wearable device, an in-vehicle device, a smart home device, and / or a smart city device. The embodiments of the present application do not impose any special restrictions on the specific type of the electronic device 100.

[0364] The electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0365] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0366] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0367] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0368] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0369] In an embodiment of the present application, the processor 110 is configured to, in response to initiating a scanning function of a code scanning application, capture a first original image using first camera parameters at the hardware layer, the first original image including a barcode. The processor 110 is further configured to generate a first target image based on the first original image at the HAL layer, and concurrently execute an action of determining the first target camera parameters based on the first original image at the HAL layer. The processor 110 is further configured to transmit the first target camera parameters to the hardware layer at the HAL layer. The processor 110 is further configured to capture a second original image using the first target camera parameters at the hardware layer, the second original image including a barcode. The processor 110 is further configured to generate a second target image based on the second original image at the HAL layer. The processor 110 is further configured to obtain, through the code scanning application, information carried by the barcode in the second target image. Specifically, the processor 110 is configured to execute any embodiment of the code scanning method provided in the embodiments of the present application.

[0370] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0371] It is understood that the interface connection relationship between the modules illustrated in the embodiments of the present application is merely an illustrative illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.

[0372] The charging management module 140 is configured to receive charging input from a charger. The charger can be either a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also provide power to the electronic device via the power management module 141.

[0373] The power management module 141 is used to connect the battery 142 , the charging management module 140 and the processor 110 .

[0374] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0375] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0376] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0377] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.) or displays an image or video through the display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0378] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0379] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-CDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0380] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0381] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD). The display screen panel can also be made of an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniLED, a microLED, a micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than one.

[0382] In an embodiment of the present application, the display screen 194 is used to display the interface of the code scanning application, such as the interface for starting the code scanning function and the interface after starting the code scanning function.

[0383] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0384] The ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image. The ISP can also perform algorithmic optimization on image noise, brightness, and other factors. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 193.

[0385] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0386] In the embodiment of the present application, the camera module including the camera 193 can be used to collect raw image data.

[0387] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0388] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. This allows electronic device 100 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0389] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU can enable intelligent cognitive applications in electronic device 100, such as image recognition, face recognition, speech recognition, and text comprehension.

[0390] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0391] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0392] The electronic device 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0393] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110.

[0394] The speaker 170A, also called a "speaker", is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or listen to hands-free calls through the speaker 170A.

[0395] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or a voice message, the user can place the receiver 170B close to the ear to hear the voice.

[0396] Microphone 170C, also known as "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the source of sound, realize directional recording function, etc.

[0397] The headphone jack 170D is used to connect a wired headphone and can be the USB interface 130 or a 3.5mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0398] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A can be located on display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. Capacitive pressure sensors can include at least two parallel plates made of conductive material.

[0399] The gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used for navigation and somatosensory game scenes.

[0400] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates the altitude using the air pressure value measured by the air pressure sensor 180C to assist in positioning and navigation.

[0401] The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip case. In some embodiments, when the electronic device 100 is a flip phone, the electronic device 100 can detect the opening and closing of the flip cover based on the magnetic sensor 180D. Based on the detected opening and closing status of the case or flip cover, features such as automatic unlocking of the flip cover can be configured.

[0402] Accelerometer 180E can detect the magnitude of acceleration of electronic device 100 in all directions (generally three axes). It can also detect the magnitude and direction of gravity when electronic device 100 is stationary. It can also be used to identify the electronic device's posture, enabling applications such as switching between landscape and portrait modes and pedometers.

[0403] The distance sensor 180F is used to measure distance. The electronic device 100 can measure distance using infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 can use the distance sensor 180F to measure distance to achieve fast focusing.

[0404] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector such as a photodiode.

[0405] Ambient light sensor 180L is used to sense ambient light brightness. Electronic device 100 can adaptively adjust the brightness of display screen 194 based on the perceived ambient light. Ambient light sensor 180L can also be used to automatically adjust white balance when taking photos. Ambient light sensor 180L can also work with proximity light sensor 180G to detect whether electronic device 100 is in a pocket to prevent accidental touches.

[0406] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0407] The temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy.

[0408] The touch sensor 180K is also called a "touch panel." The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen." The touch sensor 180K is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor can transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operations can be provided via the display screen 194. In other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 100, in a location different from that of the display screen 194.

[0409] For example, the touch sensor 180K can detect a user's touch operation, which is used to start a code scanning function, which requires starting a camera to be implemented.

[0410] The bone conduction sensor 180M can obtain a vibration signal. In some embodiments, the bone conduction sensor 180M can obtain a vibration signal of a vibrating bone in a human vocal part.

[0411] The buttons 190 include a power button, a volume button, and the like. The buttons 190 may be mechanical buttons or touch buttons. The electronic device 100 may receive key inputs and generate key signal inputs related to user settings and function control of the electronic device 100.

[0412] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts and can also be used for touch vibration feedback.

[0413] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power level changes, messages, missed calls, notifications, etc.

[0414] The SIM card interface 195 is used to connect a SIM card.

[0415] It is understandable that in order to implement the code scanning method provided in the embodiment of the present application, the electronic device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiment disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of this application.

[0416] An embodiment of the present application provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory, and the processor executes the computer program to implement the above-mentioned code scanning method.

[0417] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0418] An embodiment of the present application further provides a computer program product, including a computer program. When the computer program runs on a computer, the computer can implement the steps in the above-mentioned various method embodiments.

[0419] The present application also provides a chip system, comprising a processing circuit interface circuit, the interface circuit configured to receive code instructions and transmit them to the processing circuit, and the processing circuit configured to execute the code instructions so that the chip system implements the steps of any method embodiment of the present application. The chip system may be a single chip or a chip module composed of multiple chips.

[0420] Optionally, there may be one or more processors in the chip system. The processor may be implemented in hardware or software. When implemented in hardware, the processor may be a logic circuit, an integrated circuit, etc. When implemented in software, the processor may be a general-purpose processor implemented by reading software code stored in a memory.

[0421] Optionally, the memory in the chip system may be one or more. The memory may be integrated with the processor or may be provided separately from the processor, which is not limited in the embodiments of the present application. For example, the memory may be a non-transient processor, such as a read-only memory (ROM), which may be integrated with the processor on the same chip or provided on different chips. The embodiments of the present application do not specifically limit the type of memory or the configuration of the memory and the processor.

[0422] Exemplarily, the chip system can be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD) or other integrated chips.

[0423] It should be understood that each step in the above method embodiment can be completed by hardware integrated logic circuits in a processor or by software instructions. The method steps disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware processor, or by a combination of hardware and software modules in a processor.

[0424] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described herein are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0425] Those skilled in the art will appreciate that all or part of the process steps in the above-described method embodiments can be implemented by a computer program instructing the relevant hardware. The program can be stored in a computer-readable storage medium. When executed, the program can include the process steps in the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

[0426] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A code scanning method, characterized in that: The method is applied to an electronic device, which includes: a code scanning application, a hardware abstraction layer HAL layer, and a hardware layer. The method includes: In response to starting the code scanning function of the code scanning application, capturing a first original image using first camera parameters through the hardware layer, where the first original image includes a barcode; generating a first target image based on the first original image by the HAL layer, and concurrently performing an action of determining first target camera parameters based on the first original image by the HAL layer; Passing the first target camera parameter to the hardware layer through the HAL layer; Acquire a second original image using the first target camera parameters through the hardware layer, where the second original image includes a barcode; generating a second target image based on the second original image through the HAL layer; The information carried by the barcode in the second target image is obtained through the barcode scanning application.

2. The method according to claim 1, characterized in that After generating a first target image based on the first original image by the HAL layer and determining first target camera parameters based on the first original image by the HAL layer in parallel, the method further includes: Delivering the first target image to the code scanning application through the HAL layer; Recognizing the barcode in the first target image by the barcode scanning application; When the barcode in the first target image is recognized by the barcode scanning application, the camera parameters of the electronic device are adjusted to the first target camera parameters through the hardware layer.

3. The method according to claim 2, characterized in that The transmitting the first target image to the code scanning application through the HAL layer specifically includes: In response to generating the first target image based on the first original image through the HAL layer, sending the first target image to the code scanning application through the HAL layer; The transmitting the first target camera parameter to the hardware layer through the HAL layer specifically includes: In response to analyzing the first raw image through the HAL layer, first target camera parameters are delivered to the hardware layer through the HAL layer.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: Acquiring, through the HAL layer, second feature information of the barcode in the second original image, wherein the action of acquiring, through the HAL layer, the second feature information of the barcode in the second original image and the action of generating, through the HAL layer, a second target image based on the second original image are performed in parallel; The information carried by the barcode in the second target image obtained by the barcode scanning application specifically includes: The information carried by the barcode in the second target image is obtained through the barcode scanning application according to the second feature information.

5. The method according to claim 4, characterized in that The second feature information includes a first feature point of the barcode in the second original image, where the first feature point is in a distorted state. The information carried by the barcode in the second target image obtained by the barcode scanning application based on the second feature information specifically includes: performing correction processing on the first feature point to obtain the corrected first feature point; Information carried by the barcode in the second target image is acquired based on the corrected first feature points.

6. The method according to any one of claims 1 to 5, characterized in that Determining the first target camera parameter based on the first original image through the HAL layer specifically includes: Acquire first feature information of the barcode in the first original image through the HAL layer; The first target camera parameters are determined according to the first feature information of the barcode in the first original image through the HAL layer.

7. The method according to claim 6, characterized in that Before acquiring the first feature information of the barcode in the first original image through the HAL layer, the method further includes: Cropping the first original image through the HAL layer to obtain a first cropped image, where the first cropped image includes a barcode; The first feature information of the barcode in the first original image obtained through the HAL layer specifically includes: The first feature information of the barcode in the first original image is acquired through the HAL layer based on the first original image and the first cropped image.

8. The method according to claim 7, characterized in that The cropping of the first original image through the HAL layer specifically includes: Determining, by the HAL layer, a first a priori frame from a preset a priori frame based on feature information of a barcode in a third original image, where the third original image is acquired before the first original image; The HAL layer is used to crop the area where the first priori frame is located in the first original image to obtain the first cropped image.

9. The method according to any one of claims 1 to 8, characterized in that The first target camera parameter includes a first target zoom ratio, and determining the first target camera parameter based on the first original image through the HAL layer specifically includes: Determine, by the HAL layer, a first proportion of the barcode in the second original image; The first target zoom ratio is determined by the HAL layer according to the first proportion. When the position of the electronic device does not change, the proportion of the barcode in the original image captured using the first target zoom ratio is greater than the first proportion.

10. The method according to any one of claims 1 to 9, characterized in that The first target camera parameter includes a first target exposure parameter, and determining the first target camera parameter based on the first original image through the HAL layer specifically includes: Determine, by the HAL layer, a first grayscale value of the entire or partial area where the barcode in the second original image is located; The first target exposure parameter is determined according to the first grayscale value through the HAL layer.

11. The method according to any one of claims 1 to 10, characterized in that The first target camera parameter includes a first target focus parameter, and determining the first target camera parameter based on the first original image through the HAL layer specifically includes: Determine, by the HAL layer, a first coordinate of a center of a barcode in the second original image; The first target focus parameter is determined according to the first coordinate through the HAL layer.

12. The method according to any one of claims 1 to 11, characterized in that The method further comprises: Acquire the third original image using the first camera parameters through the hardware layer; Generate a third target image based on the third original image through the HAL layer; discarding the third target image by the code scanning application; The third original image is acquired after the first original image is acquired and before the second original image is acquired.

13. The method according to any one of claims 1 to 11, characterized in that The method further comprises: The first target image is discarded by the code scanning application.

14. An electronic device, characterized in that: The electronic device comprises a memory, a processor, and computer instructions stored in the memory, and the processor executes the computer program to implement the method according to any one of claims 1 to 13.

15. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.

16. A computer program product, characterized in that The computer program product comprises computer instructions, which implement the method according to any one of claims 1 to 13 when executed by a processor.

Images